I am trying to return a matrix of integers in a function and I decided to go with a typdef form of matrix. But when I run the project with a function that return a matrix size of 1500 by 1500, the compilation crashed after the project was built. Then I tried to work with different matrix sizes and when I compiled a the code I pasted here with a smaller size (150) for defined Matrix, the problem was solved. This is what I have tested with no problem.
typedef int Matrix[150][150];
int main(){
Matrix mat;
for(int i=0;i<13;i++){
for(int j=0;j<13;j++){
mat[i][j]=i;
}
}
cout << mat[10][11];
return 0;
}
The size of 1500 by 1500 seems very small and I cannot figure out what is the problem it is causing.
Here is the error image:
That matrix gets allocated on the stack, which is only few MB by default.
1500*1500*4 takes up about 9MB. Large arrays like that are best allocated on the heap (new/delete).
A 1500 x 1500 matrix of ints would be nearly 9MB with 32-bit ints or nearly 18MB with 64-bit ints. That's an enormous stack allocation, and you'er probably hitting a compiler or environment limit. There may be some build-time flags that could address the issue, but a more reasonable solution would be to allocate the object on the heap with new
You're probably running out of stack space - 1500*1500*sizeof(int) is roughly 9 megabytes on a 32 bit system, for example. Use an std::vector or such (it allocates from the heap) or else look up the necessary switch for your compiler to increase your stack size...
Related
I wrote the following C++ code. It is compiled on Windows 8 with MinGW (from msys). If I run it Windows will stop it and give a stack overflow error (C00000FD). What's the problem?
#include <iostream>
using namespace std;
class Test{
public:
int txt[1000];
};
int main(){
Test a[1000];
return 0;
}
What should I do, say, if I want to store a picture the size of 1920*1080? It'll be 1920*1080*4 bytes.
I believe that the default stack size in Windows is 1MB. As you are allocating 1000^2 ints, each of which is 4 bytes large, you trying to put more on the stack than it can hold.
Each Test object contains 1000 integers, likely clocking in at about 4kb each.
In main you are creating an array of 1000 objects for a total of 4MB. Your stack can't hold 4 megs.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms686774%28v=vs.85%29.aspx says a common default is 1MB.
Note that
std::vector<Test> a(1000);
Will probably work just fine. std::vector does not store its contents on the stack like a local array does.
The Test object is at least 4000 bytes in size (depending on your platform's int size). You are trying to create an array of 1000 Test objects which will be 4,000,000 bytes or nearly 4 MB. This almost certainly exceeds the default stack size for your program. You could possibly change this with some compiler options, but the question should really be what are you trying to do?
You should store large objects on the heap instead of the stack.
You can in fact change the default stack size with the following option in MinGW according to this answer:
gcc -Wl,--stack,N
But again, the better thing to do is to not store your large objects on the stack.
I need huge array in C to store some data. The thing that i am working on is related to DNA sequencing. I am using Visual Studio 2013.
Firstly, I've tried with a global static variable like
static oligo SPECTRUM[C1][C2]
Where oligo structure contains eight integers, and C1 is 100000 and C2 500.
But visual said that the array is to large. Then I asked Google, and he said that's good idea to use vectors. So i switched to these by replacing code above with a code below
static std::vector<std::vector<oligo>> SPECTRUM;
It was said that is a nice thing to resize vector before using, so i did:
SPECTRUM.resize(C1);
for (int i = 0; i < C1; i++)
{
SPECTRUM[i].resize(C2);
}
but now I am having runtime exception throwed during execution of above code (resizing)
An unhandled exception of type 'System.Runtime.InteropServices.SEHException' occurred in ConsoleApplication1.exe
in file xmemory0. Visual shows the exception is throwed here
else if (((size_t)(-1) / sizeof (_Ty) < _Count)
|| (_Ptr = ::operator new(_Count * sizeof (_Ty))) == 0)
_Xbad_alloc(); // report no memory
I want you to know also, that I have 4 GB RAM avaiable on my computer, and I estimate that my program shouldn't use more then 1 GB RAM.
Each oligo will consume 32 bytes. That means that if C1 is "around 100k", and C2 is bigger than about 600, the array will consume an entire 2 GB.
First are you sure you need all that memory available in your heap(ram)?
-You can do you calculations in chunks, allocate a chunk work on it and free it.
-You can use a file to store all your data, and load chunks of the file for your calculations.
If you need many GB of memory, it's not good to allocate it all at once in the heap, you never know there will be enough left.
I doubt there is a simple solution to this problem, given the values that you are dealing with, you will need more memory or at the very least more address space (this is "the addressable region of memory"). The easiest solution would be to go with an OS that is 64-bit - you may also need to get more RAM, but the first step is to allow the processor to address all the locations in the matrix - and with 32 bits, your limit for C2 becomes around 600, if C1 is 100k. And that assumes there are absolutely no other usage of memory - which unfortunately isn't typically true. The first few megabytes are reserved to catch "null pointer", and then the code and stack has to live somewhere. Ultimately, 100k x 500 seems unlikely to fit, even if the total size allows this much.
The other option is to use a "sparse array". Often when working with large matrices, there is a common value that is in "most places", and only some positions in the large matrix has a "different value". In these cases, you can use a method where you check if the data is present, and if so, use the value, otherwise use the default. You can use for example std::map as the storage container, and use the find method to see if the data is present.
I would suggest to address the question other way.
Make a Linked list (refer to data structure concept) for each and every element of the array as Node and get it linked. An pointer would be sufficient for accessing current node.
Yes mechanism function has to be written for traversing linked list, but will help to create such big arrays in the current target operating system instead of shifting to 64 Bit.
You should thry this:
static oligo *spectrum[C1];
for(int i = 0; i < C2; ++i)
{
spectrum[i] = new oligo[C2];
if (spectrum[i] == nullptr)
{
fprintf(stderr, "failed to allocate the array for i=%d.\n", i);
fflush(stderr);
}
}
this will tell you, how much memory are you allowed to allocate and what is your memory limit.
There may be some linker option to control this limit...
I am curious whether it is possible to determine the maximum size that an array can have in C++.
#include <iostream>
using namespace std;
#define MAX 2000000
int main()
{
long array[MAX];
cout << "Message" << endl;
return 0;
}
This compiles just fine, but then segfaults as soon as I run it (even though array isn't actually referenced). I know it's the array size too because if I change it to 1000000 it runs just fine.
So, is there some define somewhere or some way of having #define MAX MAX_ALLOWED_ARRAY_SIZE_FOR_MY_MACHINE_DEFINED_SOMEWHERE_FOR_ME?
I don't actually need this for anything, this question is for curiosity's sake.
There isn't a way to determine this statically, because the actual limit depends on how much stack space your thread has been given. You could create a new thread, give it 10 megabytes of stack, and you would be able to allocate a correspondingly larger local array.
The amount you can allocate on the stack also depends on how much has already been used so far.
void foo(int level){
int dummy[100];
cerr<<level<<endl;
foo(level+1);
}
Then, maybe you can multiply the last printed level with 400 bytes. Recursive calls occupy most of the stack space I guess but you can get a lower-bound. I may miss some understanding of memory management here, so open to corrections.
So this is what I got on my machine with varying dummy array size.
level array size stack total
24257 100 9702800
2597 1000 10388000
260 10000 10400000
129 20000 10320000
64 40000 10240000
25 100000 10000000
12 200000 9600000
Most variables declared inside functions are allocated on the stack, which is basically a block of memory of fixed size. Trying to allocate a stack variable larger than the size of the stack will cause a stack overflow, which is what the segfault is caused by.
Often the size of the stack is 8MB, so, on a 64-bit machine, long max[1000000] has size 8*1000000 < 8MB (the stack is "safe"), but long max[2000000] has size 8*2000000 > 8MB, so the stack overflows and the program segfaults.
On the other hand, dynamically allocating the array with malloc puts the memory into the heap, which is basically unbounded (4GB on a 32-bit machine, 17,179,869,184GB on a 64-bit machine).
Please read this to understand the limitations that are set by hardware and compiler. I don't think that you can have a MAX defined for you to use it right away.
I tried to check, what is the largest size of array, which can be created in CPP. I declared a "int" array and kept increasing the array size. After 10^9 the program started crashing, but there was a serious error for array of size 5*10^8 and more(even when program did not crash). The code used and the problem is following :
#include<iostream>
int ar[500000000];
int main()
{
printf("Here\n");
}
The above code runs successfully, if the size of array is reduced to 4*10^8 and lesser. But, for array size greater than 5*10^8, the program runs successfully but it does not print any thing, also it does not get crashed, or gave any error or warning.
Also, if the array definition is local then there is no such error, after same limit the program gets crashed. It's when using the global definition of array, the program does not get crashed nor does print anything.
Can anybody please explain the reason for this behavior. I've understood that the size of array will vary for different machines.
I've 1.2 GB of free RAM. How am I able to create the local integer array of size 4*10^8. This requires around 1.49GB, and I don't have that much free RAM.
The real question is: why are you using globals? And to make it worse, it's a static raw array?
As suggested already, the memory used to hold global variables is being overflowed (and it possibly wrote over your "Here\n" string).
If you really need that big of an array, use dynamically-allocated memory:
int main() {
int* bigArray = new int[500000000];
// ... use bigArray here
delete[] bigArray;
}
C++ inherently doesn't restrict the max limits on the array size. In this case since it is a global variable it will be outside the stack as well. The only thing I can think of is the memory limit on your machine. How much memory does your machine has? How much memory is free before running your program?
Simple question, I'm writting a program that needs to open huge image files (8kx8k) but I'm a little bit confused on how to initialize the huge arrays to hold the images in c++.
I been trying something like this:
long long SIZE = 8092*8092; ///8096*8096
double* array;
array = (double*) malloc(sizeof(double) * SIZE);
if (array == NULL)
{
fprintf(stderr,"Could not allocate that much memory");
}
But sometimes my NULL check does not catch that the array was not initialized, any idea why?
Also I can't initialize more that 2 or 3 arrays, even when running in a x64 machine with 12 GB of RAM, any idea why?
I would really wish not to have to work with sections of array instead. Any help is welcome.
Thanks.
You're not running into an array size problem. 8K*8K is merely 64M. Even 64M doubles (sizeof==8) are not an issue; that would require a mere 512 MB. Now, a 32 bit application (no matter where it's running) should be able to allocate a few of them. Not 8, because the OS typically needs to reserve some space for itself (often slightly over 2GB) and sometimes not even 3 when memory is fragmented.
The behavior of "malloc failed but didn't return NULL" is a Linux configuration bug, fixed by # echo 2 > /proc/sys/vm/overcommit_memory
malloc() does not initialize memory, it just reserves it. You will have to initialize it explicitly, e.g. via memset() from string.h:
array = (double*) malloc(SIZE * sizeof(double));
if (array) memset(array, 0, SIZE * sizeof(double));
However, in C++ you should use new instead of malloc:
double* array = new double[SIZE];
if (!array) {
cerr << "Could not allocate that much memory" << endl;
}
for (int i=0; i<SIZE; i++) array[i] = 0.0;
Regarding size: each such array is 512 MB. Are you positively sure you need double precision (which means the image has 64-bit pixel depth)? Maybe a float would suffice? That would halve the memory footprint.
You might be running into a 2GB per-process address space limit if you are running a 32bit operating system. With a few hundred MBs of system libs and other stuff, and 2 or 3 arrays of 512MB each, that will give 2GB easily. A 64bit OS would help you there.
Are you compiling your application as a 32-bit application (the default in Visual Studio, if that's what you're using), or as a 64-bit application? You shouldn't have troubles if you build it as a 64-bit app.
malloc allocates (reserves memory and returns a pointer), calloc initializes (writes all zeros to that memory).
Seems to be that you have no continuous memory block of such size (~500Mb) in C runtime heap. Instead of copying file into memory try to map image into a processes address space. You could map only necessary parts of the file.
Just as a side note: although you don't want to bother about the whole image not being in memory at once, there are reasons not to do it. Maybe think about an abstraction that allows you to keep only the currently needed chunk in memory. The program code then can be written as though ignorant of the memory issues.
I would really wish not to have to work with sections of array instead. Any help is welcome.
Have you looked into memory-mapped files?
Yep, sounds a lot like heap fragmentation, as Kirill pointed out. See also: How to avoid heap fragmentation?
i suggest using compression. decompress part of it which you need to process in your code whenever, and compress it after the part done.
2nd proposal: write code to overload memory pointer "operator+" and "operator-" so you could use non-continuous memory buffers. use smaller memory buffers make your code more stable than a continuous larger one. i had experienced it and had written some operator-overloading, see http://code.google.com/p/effoaddon/source/browse/trunk/devel/effo/codebase/addons/mem/include/mcur_i.h for the example. when i test 47G malloc()ed system memory on a x86_64, i allocated just 1G per malloc() call, so i allocated 47 memory blocks in total. EDIT: while if i tried to allocate as much as possible by using just one malloc(), i would only get 30G on a 48G system, say less than 70%, that's because larger buffer per malloc() requested, much more managemental memory consumed by the system/libc itself, you know, I called mlock() to prevent the allocated memory from being swapped out to the disk.
3rd one: try posix file mapping, map to memory per image.
Btw: call malloc() is more stable than new() though writing c++, because when memory got stressed, new() is prone to trow exceptions instead of returning NULL.