"Process is terminated due to StackOverflowException" is the error I receive when I run the code below. If I change 63993 to 63992 or smaller there are no errors. I would like to initialize the structure to 100,000 or larger.
#include <Windows.h>
#include <vector>
using namespace std;
struct Point
{
double x;
double y;
};
int main()
{
Point dxF4struct[63993]; // if < 63992, runs fine, over, stack overflow
Point dxF4point;
vector<Point> dxF4storage;
for (int i = 0; i < 1000; i++) {
dxF4point.x = i; // arbitrary values
dxF4point.y = i;
dxF4storage.push_back(dxF4point);
}
for (int i = 0; i < dxF4storage.size(); i++) {
dxF4struct[i].x = dxF4storage.at(i).x;
dxF4struct[i].y = dxF4storage.at(i).y;
}
Sleep(2000);
return 0;
}
You are simply running out of stackspace - it's not infinite, so you have to take care not to run out.
Three obvious choices:
Use std::vector<Point>
Use a global variable.
Use dynamic allocation - e.g. Point *dxF4struct = new Point[64000]. Don't forget to call delete [] dxF4struct; at the end.
I listed the above in order that I think is preferable.
[Technically, before someone else points that out, yes, you can increase the stack, but that's really just moving the problem up a level somewhere else, and if you keep going at it and putting large structures on the stack, you will run out of stack eventually no matter how large you make the stack]
Increase the stack size. On Linux, you can use ulimit to query and set the stack size. On Windows, the stack size is part of the executable and can be set during compilation.
If you do not want to change the stack size, allocate the array on the heap using the new operator.
Well, you're getting a stack overflow, so the allocated stack is too small for this much data. You could probably tell your compiler to allocate more space for your executable, though just allocating it on the heap (std::vector, you're already using it) is what I would recommend.
Point dxF4struct[63993]; // if < 63992, runs fine, over, stack overflow
That line, you're allocating all your Point structs on the stack. I'm not sure the exact memory size of the stack but the default is around 1Mb. Since your struct is 16Bytes, and you're allocating 63393, you have 16bytes * 63393 > 1Mb, which causes a stackoverflow (funny posting aboot a stackoverflow on stack overflow...).
So you can either tell your environment to allocate more stack space, or allocate the object on the heap.
If you allocate your Point array on the heap, you should be able to allocate 100,000 easily (assuming this isn't running on some embedded proc with less than 1Mb of memory)
Point *dxF4struct = new Point[63993];
As a commenter wrote, it's important to know that if you "new" memory on the heap, it's your responsibility to "delete" the memory. Since this uses array new[], you need to use the corresponding array delete[] operator. Modern C++ has a smart pointer which will help with managing the lifetime of the array.
Related
Good night
I'm learning C++ for scientific code development. Differently of my learning strategy used in python and MATLAB, I'm trying to learn C++ in old and deep way (using a book ref[1]). I'm struggling to understand pointers and dynamic memory allocation.
I came across this exercise and some doubts appeared. The exercise is a simple 2x2 matrix multiplication. The code:
#include<iostream>
//Perform C[2][2] = A[2][2]*B[2][2] Using dynamic memory allocation
int main(){
//declaring length of the matrices
int row,col;
row = 2;
col = 2;
//Allocating memory for matrice A [heap ?]
double** A, **B, **C;
A = new double* [row];
B = new double* [row];
C = new double* [row];
for (int i=0; i<row; ++i){
A[i] = new double [col];
B[i] = new double [col];
C[i] = new double [col];
}
//Performing calculation
for (int i=0; i<2; ++i){
for (int j=0; j<2; ++j){
A[i][j] += 10;
B[i][j] += 20;
C[i][j] += A[i][j]*B[i][j];
std::cout << C[i][j] << " ";
}
std::cout << std::endl;
}
//Deleting memory allocation [Is there an memory leak ? (No *=NULL)]
for (int i=0; i<2; ++i){
delete[] A[i];
delete[] B[i];
delete[] C[i];
}
delete[] A;
delete[] B;
delete[] C;
return 0;
}
My doubts are:
1 - I read about the memory division (global/stack/heap), where are they located in the hardware ? HD/RAM/CPU cache ?
2 - when I allocate an array:
int arrayS[2]; //Is this at the stack ?
int* arrayH; //Is this at the heap ?
arrayH = new int [2];
3 - In my resolution of the matrix problem. Is there a memory leak or garbage creation ? (note I didn't pointed the arrays to *=NULL to get rid of the address)
4 - Do you suggest any way to improve my code performance and efficiency ?
Thank you !
1) The global area, stack and heap are all located within your application's CPU address space. Which will mean they're in RAM, subject to virtual memory paging (in which case they may go and live on the HD for a while) and CPU caching (in which case they may go and live on the CPU for a while). But both of those things are transparent as far as the generated code is concerned.
2) yes, arrayS[2] will be on the stack (unless it's global, of course), and anything returned by new (or malloc, should you include some C code) is on the heap.
3) I can't see any leaks but if you're going to use row and col rather than repeating the 2 magic constant all over the place then do so uniformly and mark them as const so that you can't accidentally modify them.
4) in terms of cache efficiency, it may be better to do one allocation as new double[col * row] and then either to spread your pointers out throughout that block or to index as [i*col + j], assuming i indexes rows and all rows are col items long. new is otherwise permitted to spread your rows out wherever it likes across memory, which is more likely to lead to cache faults.
As to style? Well, you didn't ask, but I think cdhowie's comment is valid. Also Deduplicator's point should be read more deeply: the STL has a bunch of pieces to help make sure you don't leak memory — read up on unique_ptr and shared_ptr.
I read about the memory division (global/stack/heap), where are they located in the hardware ? HD/RAM/CPU cache ?
They can all be stored wherever your particular C++ program decides that they should be. Some local variables are likely to exist only in registers, if they have a short life span and never have a pointer taken to them.
Odds are that everything else will wind up somewhere in system RAM (which can be paged out to disk, if your process is unlucky). The difference is how they are used by the program, not where they happen to be. Where they are is an implementation detail that you don't really need to worry about.
when I allocate an array
Yes, your analysis of stack vs. heap there is correct. Your first one (int arrayS[2];) has the caveat that if you declare this variable as part of a class or struct, it could exist either place depending on how the class/struct gets created. (If the class/struct gets created on the heap then the array would be on the heap; if it gets created on the stack then on the stack.)
And of course if it's a global then it has global storage.
Do you suggest any way to improve my code performance and efficiency ?
Encapsulate the concept of a matrix inside of a class, and use std::vector instead of allocating your own arrays. The amount of indirection will be the same after compiler optimization, but you no longer have to concern yourself with managing the memory used by the arrays.
In c++ the object can be located stack and in heap.In your example int arrayS[2] is located in the stack of the funcion in which it was declared.If you want to create object in heap (a memory that is available to all the functions that exists in the program ) you must use the operator new.The pointer that keeps the address of the object allocated (in your case arrayH) is also push in the stack of the function in which it is declared
Actually memory of one process(program that is being executable) is dived in three parts:
Code Area (is the area in which the code of the program is located );
The stack area (is starts at the location where the Stack-Pointer points in memory.Stack pointer is a register that keeps every time the address of the current stack.In a program more that one stack at one time,this depends on the level of recursivity of your program)
The data area (is the global memory accessible from entire program.It is allocated in c++ only with new Operator or as global variable)
It exists also shared_memory which permits to the programmer to allocate memory available in more than one process.
**There is no memory leak in your code.But in c++ are implemented the concept of smart_pointer which is very similar with the garabage-colector from C# and Java **
Also the C++ is an OOP programming language so you definitely should write your code using classes ,inheritance,polimorfism etc.
I am trying to find the maximum memory allocated using new[]. I have used binary search to make allocation a bit faster, in order to find the final memory that can be allocated
bool allocated = false;
int* ptr= nullptr;
int low = 0,high = std::numeric_limits<int>;
while(true)
{
try
{
mid = (low + high) / 2;
ptr = new int[mid];
delete[] ptr;
allocated = true;
}
catch(Exception e)
{....}
if (allocated == true)
{
low = mid;
}else
{
high = low;
cout << "maximum memory allocated at: " << ptr << endl;
}
}
I have modified my code, I am using a new logic to solve this. My problem right now is it is going to a never ending loop. Is there any better way to do this?
This code is useless for a couple of reasons.
Depending on your OS, the memory may or may not be allocated until it is actually accessed. That is, new happily returns a new memory address, but it doesn't make the memory available just yet. It is actually allocated later when and if a corresponding address is accessed. Google up "lazy allocation". If the out-of-memory condition is detected at use time rather than at allocation time, allocation itself may never throw an exception.
If you have a machine with more than 2 gigabytes available, and your int is 32 bits, alloc will eventually overflow and become negative before the memory is exhausted. Then you may get a bad_alloc. Use size_t for all things that are sizes.
Assuming you are doing ++alloc and not ++allocation, it shouldn't matter what address it uses. if you want it to use a different address every time then don't delete the pointer.
This is a particularly bad test.
For the first part you have undefined behaviour. That's because you should only ever delete[] the pointer returned to you by new[]. You need to delete[] pvalue, not value.
The second thing is that your approach will be defragmenting your memory as you're continuously allocating and deallocating contiguous memory blocks. I imagine that your program will understate the maximum block size due to this fragmentation effect. One solution to this would be to launch instances of your program as a new process from the command line, setting the allocation block size as a parameter. Use a divide and conquer bisection approach to attain the maximum size (with some reliability) in log(n) trials.
This question already has answers here:
Segmentation fault on large array sizes
(7 answers)
Closed 4 years ago.
I'm converting a program from fortran to C++.
My code seems to run fine until I add this array declaration:
float TC[100][100][100];
And then when I run it I get a segmentation fault error. This array should only take up 8Mb of memory and my machine has 3 Gb. Is there a problem with this declaration? My c++ is pretty rusty.
That array is about 4 megabyte large. If this definition is inside a function (as local variable), then the compiler tries to store it on the stack, which on most systems cannot grow that large.
The Fortran compiler probably allocated it statically (Fortran routines are not allowed to be called recursively unless explicitly marked as recursive, so static allocation for local variables works there for non-recursive functions), and therefore the error doesn't occur there.
A simple fix would be to explicitly declare the variable static, assuming the Fortran function was not declared recursive. However this may bite you later, if you ever try to call that function recursively from a revised version. So a better solution would probably be to allocate it dynamically. However that costs extra time and therefore depending on the nature of the code, may hurt your performance too much (Fortran code quite often is numerical code where performance matters).
If you choose to make the array static, you can build in a protection against accidental recursive calls:
void yourfunction()
{
static bool active;
static float TC[100][100][100];
assert(!active);
active = true;
// your code
active = false;
}
I'm guessing TC is being allocated as an auto local variable. This means it's being stored on the stack. You don't get 4mb of stack memory, so it's causing a stack overflow.
To solve it, use dynamic allocation with a structured container or new.
This looks like a stack-based declaration. Try allocating from the heap (i.e. using the new operator).
If you are declaring it inside of a function, as a local variable, it may be that you stack is not big enough to fit the array. You may try to allocate in the heap, with new or malloc(), or, if your design allows, make it a global variable.
In C++ the stack has a limited amount of space. MSVC defaults this size to 1MB. If the stack uses more than 1MB it will segfault or stackoverflow or something. You will have to move that structure to dynamic memory. To move it to dynamic memory, you want something like this:
typedef float (bigarray)[100][100][100];
bigarray& TC() {
static bigarray* ptr = NULL;
if (ptr == NULL) {
ptr = new float[100][100][100];
for(int j=0; j<100; j++) {
ptr[j] = new float[100][100];
for(int i=0; i<100; i++)
ptr[j][i] = new float[100];
}
}
return *ptr;
}
That will allocate the structure in dynamic memory the first time it is accessed, as a jagged array. You can get more performance out of a rectangle array, but you have to change types:
typedef std::vector<std::array<std::array<float, 100>, 100> bigarray;
bigarray TC(100);
According to http://cs.nyu.edu/exact/core/doc/stackOverflow.txt, gcc/linux defaults the stack size to 8MB, which isn't big enough for your structure and int main() If you really want to, MSVC has flags to increase the stack sizes up to 32MB. Linux has a ulimit command to increase the stack size up to 32MB.
I'm making an application that is going to be using many dynamically created objects (raytracing). Instead of just using [new] over and over again, I thought I'd just make a simple memory system to speed things up. Its very simple at this point, as I don't need much.
My question is: when I run this test application, using my memory manager uses the correct amount of memory. But when I run the same loop using [new], it uses 2.5 to 3 times more memory. Is there just something I'm not seeing here, or does [new] incur a huge overhead?
I am using VS 2010 on Win7. Also I'm just using the Task Manager to view the process memory usage.
template<typename CLASS_TYPE>
class MemFact
{
public:
int m_obj_size; //size of the incoming object
int m_num_objs; //number of instances
char* m_mem; //memory block
MemFact(int num) : m_num_objs(num)
{
CLASS_TYPE t;
m_obj_size = sizeof(t);
m_mem = new char[m_obj_size * m_num_objs);
}
CLASS_TYPE* getInstance(int ID)
{
if( ID >= m_num_objs) return 0;
return (CLASS_TYPE*)(m_mem + (ID * m_obj_size));
}
void release() { delete m_mem; m_mem = 0; }
};
/*---------------------------------------------------*/
class test_class
{
float a,b,c,d,e,f,g,h,i,j; //10 floats
};
/*---------------------------------------------------*/
int main()
{
int num = 10 000 000; //10 M items
// at this point we are using 400K memory
MemFact<test_class> mem_fact(num);
// now we're using 382MB memory
for(int i = 0; i < num; i++)
test_class* new_test = mem_fact.getInstance(i);
mem_fact.release();
// back down to 400K
for(int i = 0; i < num; i++)
test_class* new_test = new test_class();
// now we are up to 972MB memory
}
There is a minimum size for a memory allocation, depending on the CRT you are using. Often that's 16 bytes. Your object is 12 bytes wide (assuming x86), so you're probably wasting at least 4 bytes per allocation right there. The memory manager also has it's own structures to keep track of what memory is free and what memory is not -- that's not free. Your memory manager is probably much simplier (e.g. frees all those objects in one go) which is inherently going to be more efficient than what new does for the general case.
Also keep in mind that if you're building in debug mode, the debugging allocator will pad both sides of the returned allocation with canaries in an attempt to detect undefined behavior. That'll probably put you over the 16 byte boundary and into the next one -- probably a 32 byte allocation, at least. That'll be disabled when you build in release mode.
Boy, I sure hope that nobody wants to allocate any non-PODs from your memory manager. Or objects of dynamic size. And doesn't mind instantiating it for every type. Or creating as many as they like all at once. Or having their lifetime be longer than the MemFact.
In fact, there is a valid pattern known as an Object Pool, which is similar to yours but doesn't suck. The simple answer is that operator new is required to be ultra flexible- it's objects must live forever until delete is called- and their destructor must be called too, and they must all have completely separate, independent lifetimes. It must be able to allocate variable-size objects, and of any type, at any time. Your MemFact meets none of these requirements. The Object Pool also has less requirements, and is significantly faster than regular new because of it, but it also doesn't completely fail on all the other fronts.
You're trying to compare an almost completely rotten apple with an orange.
When I run my program with 1 array, like this:
int a[430][430];
int i, j, i_r0, j_r0;
double c, param1, param2;
int w_far = 0,h_far = 0;
char* magic_num1 = "";
it's good!
But, when I write:
int a[430][430];
int i, j, i_r0, j_r0;
int nicky[430][430]; // Added line
double c, param1, param2;
int w_far = 0,h_far = 0;
char* magic_num1 = "";
the program not run with the error: "stack overflow"!
I don't know how to solve it!
You need to either increase the stack space (how that is done depends on your platform), or you need to allocate the array from the heap, or even better, use std::vector instead of an array.
You're trying to allocate ~1.48 MB of stuff on the stack1, on your system (and not only on it) that's too much.
In general, the stack is not made for keeping big objects, you should put them in the heap instead; use dynamic allocation with new or std::vector, or, even better suited in your case, boost::multi_array.
1. Assuming 32 bit ints.
A proper solution is to use heap, but also note that you'll likely find that changing to:
short a[430][430];
short nicky[430][430]; // Added line
fixes the overflow, depending on your platform. So if 'short', or 'unsigned short' is big enough, this might be an option.
In fact, even when using the heap, consider carefully the array type to reduce memory footprint for a large array.
Local variables are allocated to "stack", which is a storage space used to several purposes and limited to a certain size.
Usually you can declare variables up to several kilobytes, but when you want to use more memory, usually suggested to use "heap", which can be allocated by new operator or std::vector.
std::vector is an alternate for traditional arrays, and its data is safely stored in heap.
To avoid stack overflow, allocate the arrays in the heap.
If one uses C, then allocating an array of size n in the heap can be done by e.g.
int* A = (int*) malloc(n*sizeof(int));
But you must remeber to free that memory when no longer needed with
free(A);
to avoid memory leak.
Equivalently in C++:
int* A = new int[n];
and free with
delete [] A;
This site was helpful.