I need huge array in C to store some data. The thing that i am working on is related to DNA sequencing. I am using Visual Studio 2013.
Firstly, I've tried with a global static variable like
static oligo SPECTRUM[C1][C2]
Where oligo structure contains eight integers, and C1 is 100000 and C2 500.
But visual said that the array is to large. Then I asked Google, and he said that's good idea to use vectors. So i switched to these by replacing code above with a code below
static std::vector<std::vector<oligo>> SPECTRUM;
It was said that is a nice thing to resize vector before using, so i did:
SPECTRUM.resize(C1);
for (int i = 0; i < C1; i++)
{
SPECTRUM[i].resize(C2);
}
but now I am having runtime exception throwed during execution of above code (resizing)
An unhandled exception of type 'System.Runtime.InteropServices.SEHException' occurred in ConsoleApplication1.exe
in file xmemory0. Visual shows the exception is throwed here
else if (((size_t)(-1) / sizeof (_Ty) < _Count)
|| (_Ptr = ::operator new(_Count * sizeof (_Ty))) == 0)
_Xbad_alloc(); // report no memory
I want you to know also, that I have 4 GB RAM avaiable on my computer, and I estimate that my program shouldn't use more then 1 GB RAM.
Each oligo will consume 32 bytes. That means that if C1 is "around 100k", and C2 is bigger than about 600, the array will consume an entire 2 GB.
First are you sure you need all that memory available in your heap(ram)?
-You can do you calculations in chunks, allocate a chunk work on it and free it.
-You can use a file to store all your data, and load chunks of the file for your calculations.
If you need many GB of memory, it's not good to allocate it all at once in the heap, you never know there will be enough left.
I doubt there is a simple solution to this problem, given the values that you are dealing with, you will need more memory or at the very least more address space (this is "the addressable region of memory"). The easiest solution would be to go with an OS that is 64-bit - you may also need to get more RAM, but the first step is to allow the processor to address all the locations in the matrix - and with 32 bits, your limit for C2 becomes around 600, if C1 is 100k. And that assumes there are absolutely no other usage of memory - which unfortunately isn't typically true. The first few megabytes are reserved to catch "null pointer", and then the code and stack has to live somewhere. Ultimately, 100k x 500 seems unlikely to fit, even if the total size allows this much.
The other option is to use a "sparse array". Often when working with large matrices, there is a common value that is in "most places", and only some positions in the large matrix has a "different value". In these cases, you can use a method where you check if the data is present, and if so, use the value, otherwise use the default. You can use for example std::map as the storage container, and use the find method to see if the data is present.
I would suggest to address the question other way.
Make a Linked list (refer to data structure concept) for each and every element of the array as Node and get it linked. An pointer would be sufficient for accessing current node.
Yes mechanism function has to be written for traversing linked list, but will help to create such big arrays in the current target operating system instead of shifting to 64 Bit.
You should thry this:
static oligo *spectrum[C1];
for(int i = 0; i < C2; ++i)
{
spectrum[i] = new oligo[C2];
if (spectrum[i] == nullptr)
{
fprintf(stderr, "failed to allocate the array for i=%d.\n", i);
fflush(stderr);
}
}
this will tell you, how much memory are you allowed to allocate and what is your memory limit.
There may be some linker option to control this limit...
Related
I am currently having a problem with declaring or filling a large array with data because I get a dialog box saying "Out of memory", originating from CMemoryException.
I am trying to create an array or vector (tried both) with around 50000 elements of an object, where sizeof(MyObjectClass) returns around 37000 bytes.
If I try to just fill up a vector or a CArray element by element, then I get around to filling with somewhere near 16000 elements before getting the Out Of Memory exception. That should be close to 600MBs?
I have 8GB RAM on the machine and only 4GB are being used according to Windows Task Manager. So the amount of physical RAM should not impose a problem. I am running C++ MFC in Visual Studio 2010, 32-bit.
Also if I try to write
MyObjectClass* heaparray = new MyObjectClass[50000];
then I immediately get that very same Out of memory error, on that very row.
Any ideas?
Thank You in advance!
UPDATE:
I have also tried to simply create a TestStruct with the fields:
struct TestStruct
{
long long field1;
GUID field2;
GUID field3;
GUID field4;
TCHAR field5[256];
TCHAR field6[4];
TCHAR field7[258];
TCHAR field8[1026];
TCHAR field9[258];
TCHAR field10[16386];
TCHAR field11[258];
};
TestStruct* heapArr = new TestStruct[50000];
Still the same...I get a "Out of Memory" exception when executing the last line of code.
Isn't one of the great things with the heap supposed to be possibility to be limited only by RAM (more or less) when handling big data. And yet...since it crashes already at 600MB of allocated space I cannot agree that that is very big data either...or should I? :/
This is a fun one. Both Vectors and arrays are stored contiguously in memory as stated here.
You are not only looking for 1850000000 bytes (1.72295 gigabytes) in memory, but one unbroken chunk of memory that big. That will be hard to find. If you switch to a different data structure that does not do contiguous storage (say a linked list) then you may be able to store that much.
Note: that will also make each object just a bit bigger.
What would be best would be to see if there is any way to just buffer the objects; load only the ones you will update and load the others on the fly when you need them. I have my doubts that you are doing cpu operations on more than one at a time. If you do it right (with threading most likely) you won't even suffer any slows from reading/writing them.
More information about what you are working on would be helpful. There may even be a way to just have an array filled with a type identifier, if your object has less than 2,147,483,647 (size of int) variations. You could store an array of integers that the class could be generated from (a toHash and fromHash that would be 50000 * 4 bytes = 195.312 kilobytes), that may work for you too. Again, it depends on what you are working on.
I will try to expand on #user1884803's answer:
Don't use a pointer to an array. Even Visual Studio 2010 has <vector>. But see next point.
Don't use a vector either... Specially if you really want to read all your MyObjectClass objects in RAM. As the other answer said, even if you have 4Gbytes free, you probably don't have 1.7Gbytes of contiguous free memory.
So, if you really, really, want to read all your objects in RAM (because the processing you want to do on them is non-linear, or needs many records at the same time in memory), use a std::list<MyObjectClass> or, if you need a "key" to access each record, use a std::map<KeyType, MyObjectClass>. BUT...
You really should try not reading 1.8Gbytes of objects to RAM. Even if you have that much RAM lying around unused, it's just not a good practice. If you can, read each object from the database, process it, and write it back to the database discarding the used object, not accumulating the whole thing in RAM. If you need and if it improves your speed, you can save part of it in a std::list, std::map, or even in a std::vector, and on demand refresh other parts of the objects from the database.
That way, your program would go from:
if( cmd.Open() ) {
do {
MyObjectClass obj = cmd.Read(); // whatever is needed to read the object from the db
vectorOfObjects.push_back(obj); // or list, or map...
} while( cmd.MoveNext() );
}
for( std::vector<MyObjectClass>::iterator p = vectorOfObjects.begin(), e = vectorOfObjects.end(); p != e; ++p ) {
// process *p
}
for( std::vector<MyObjectClass>::iterator p = vectorOfObjects.begin(), e = vectorOfObjects.end(); p != e; ++p ) {
cmd.Save(*p); // see reading above, but for saving...
}
to something like
if( cmd.Open() ) {
do {
MyObjectClass obj = cmd.Read();
// JUST PROCESS obj here and go to next
cmd.Save(obj); // or whatever
} while( cmd.MoveNext() );
}
I've got a very basic application that boils down to the following code:
char* gBigArray[200][200][200];
unsigned int Initialise(){
for(int ta=0;ta<200;ta++)
for(int tb=0;tb<200;tb++)
for(int tc=0;tc<200;tc++)
gBigArray[ta][tb][tc]=new char;
return sizeof(gBigArray);
}
The function returns the expected value of 32000000 bytes, which is approximately 30MB, yet in the Windows Task Manager (and granted it's not 100% accurate) gives a Memory (Private Working Set) value of around 157MB. I've loaded the application into VMMap by SysInternals and have the following values:
I'm unsure what Image means (listed under Type), although irrelevant of that its value is around what I'm expecting. What is really throwing things out for me is the Heap value, which is where the apparent enormous size is coming from.
What I don't understand is why this is? According to this answer if I've understood it correctly, gBigArray would be placed in the data or bss segment - however I'm guessing as each element is an uninitialised pointer it would be placed in the bss segment. Why then would the heap value be larger by a silly amount than what is required?
It doesn't sound silly if you know how memory allocators work. They keep track of the allocated blocks so there's a field storing the size and also a pointer to the next block, perhaps even some padding. Some compilers place guarding space around the allocated area in debug builds so if you write beyond or before the allocated area the program can detect it at runtime when you try to free the allocated space.
you are allocating one char at a time. There is typically a space overhead per allocation
Allocate the memory on one big chunk (or at least in a few chunks)
Do not forget that char* gBigArray[200][200][200]; allocates space for 200*200*200=8000000 pointers, each word size. That is 32 MB on a 32 bit system.
Add another 8000000 char's to that for another 8MB. Since you are allocating them one by one it probably can't allocate them at one byte per item so they'll probably also take the word size per item resulting in another 32MB (32 bit system).
The rest is probably overhead, which is also significant because the C++ system must remember how many elements an array allocated with new contains for delete [].
Owww! My embedded systems stuff would roll over and die if faced with that code. Each allocation has quite a bit of extra info associated with it and either is spaced to a fixed size, or is managed via a linked list type object. On my system, that 1 char new would become a 64 byte allocation out of a small object allocator such that management would be in O(1) time. But in other systems, this could easily fragment your memory horribly, make subsequent new and deletes run extremely slowly O(n) where n is number of things it tracks, and in general bring doom upon an app over time as each char would become at least a 32 byte allocation and be placed in all sorts of cubby holes in memory, thus pushing your allocation heap out much further than you might expect.
Do a single large allocation and map your 3D array over it if you need to with a placement new or other pointer trickery.
Allocating 1 char at a time is probably more expensive. There are metadata headers per allocation so 1 byte for a character is smaller than the header metadata so you might actually save space by doing one large allocation (if possible) that way you mitigate the overhead of each individual allocation having its own metadata.
Perhaps this is an issue of memory stride? What size of gaps are between values?
30 MB is for the pointers. The rest is for the storage you allocated with the new call that the pointers are pointing to. Compilers are allowed to allocate more than one byte for various reasons, like to align on word boundaries, or give some growing room in case you want it later. If you want 8 MB worth of characters, leave the * off your declaration for gBigArray.
Edited out of the above post into a community wiki post:
As the answers below say, the issue here is I am creating a new char 200^3 times, and although each char is only 1 byte, there is overhead for every object on the heap. It seems creating a char array for all chars knocks the memory down to a more believable level:
char* gBigArray[200][200][200];
char* gCharBlock=new char[200*200*200];
unsigned int Initialise(){
unsigned int mIndex=0;
for(int ta=0;ta<200;ta++)
for(int tb=0;tb<200;tb++)
for(int tc=0;tc<200;tc++)
gBigArray[ta][tb][tc]=&gCharBlock[mIndex++];
return sizeof(gBigArray);
}
I don't know what to think here...
We have a component that runs as a service. It runs perfectly well on my local machine, but on some other machine (on both machine RAM's are equal to 2GB) it starts to generate bad_alloc exceptions on the second and consecutive days. The thing is that the memory usage of the process stays the same at aproximately 50Mb level. The other weird thing is that by means of tracing messages we have localized the exception to be thrown from a stringstream object which does but insert no more than 1-2 Kb data into the stream. We're using STL-Port if that matters.
Now, when you get a bad_alloc exception, you think it's a memory leak. But all our manual allocations are wrapped into a smart pointer. Also, I can't understand how a stringstream object lacks memory when the whole process uses only ~50Mb (the memory usage stays approximtely constant(and sure doesn't rise) from day to day).
I can't provide you with code, because the project is really big, and the part which throws the exception really does nothing else but create a stringstream and << some data and then log it.
So, my question is... How can a memory leak/bad_alloc occur when the process uses only 50Mb memory out of 2GB ? What other wild guesses do you have as to what could possibly be wrong?
Thanks in advance, I know the question is vague etc., I'm just sort of desperate and I tried my best to explain the problem.
One likely reason within your description is that you try to allocate a block of some unreasonably big size because of an error in your code. Something like this;
size_t numberOfElements;//uninitialized
if( .... ) {
numberOfElements = obtain();
}
elements = new Element[numberOfElements];
now if numberOfElements is left uninitialized it can contain some unreasonably big number and so you effectively try to allocate a block of say 3GB which the memory manager refuses to do.
So it can be not that your program is short on memory, but that it tries to allocate more memory than it could possibly be allowed to under even the best condition.
bad_alloc doesn't necessarily mean there is not enough memory. The allocation functions might also fail because the heap is corrupted. You might have some buffer overrun or code writing into deleted memory, etc.
You could also use Valgrind or one of its Windows replacements to find the leak/overrun.
Just a hunch,
But I have had trouble in the past when allocating arrays as so
int array1[SIZE]; // SIZE limited by COMPILER to the size of the stack frame
when SIZE is a large number.
The solution was to allocate with the new operator
int* array2 = new int[SIZE]; // SIZE limited only by OS/Hardware
I found this very confusing, the reason turned out to be the stack frame as discussed here in the solution by Martin York:
Is there a max array length limit in C++?
All the best,
Tom
Check the profile of other processes on the machine using Process Explorer from sysinternals - you will get bad_alloc if memory is short, even if it's not you that's causing memory pressure.
Check your own memory usage using umdh to get snapshots and compare usage profile over time. You'll have to do this early in the cycle to avoid blowing up the tool, but if your process's behaviour is not degrading over time (ie. no sudden pathological behaviour) you should get accurate info on its memory usage at time T vs time T+t.
Another long shot: you don't say in which of the three operations the error occurs (construction, << or log), but the problem may be memory fragmentation, rather than memory consumption. Maybe stringstream can't find a contiguous memory block long enough to hold a couple of Kb.
If this is the case, and if you exercise that function on the first day (without mishap) then you could make the stringstream a static variable and reuse it. As far as I know stringstream does not deallocate it's buffer space during its lifetime, so if it establishes a big buffer on the first day it will continue to have it from then on (for added safety you could run a 5Kb dummy string through it when it is first constructed).
I fail to see why a stream would throw. Don't you have a dump of the failed process? Or perhaps attach a debugger to it to see what the allocator is trying to allocate?
But if you did overload the operator <<, then perhaps your code does have a bug.
Just my 2 (euro) cts...
1. Fragmentation ?
The memory could be fragmented.
At one moment, you try to allocate SIZE bytes, but the allocator finds no contiguous chunk of SIZE bytes in memory, and then throw a bad_alloc.
Note: This answer was written before I read this possibility was ruled out.
2. signed vs. unsigned ?
Another possibility would be the use of a signed value for the size to be allocated:
char * p = new char[i] ;
If the value of i is negative (e.g. -1), the cast into the unsigned integral size_t will make it go beyond what is available to the memory allocator.
As the use of signed integral is quite common in user code, if only to be used as a negative value for an invalid value (e.g. -1 for a failed search), this is a possibility.
~className(){
//delete stuff in here
}
By way of example, Memory leaks can occur when you use the new operator in c++ and forget to use the delete operator.
Or, in other words, when you allocate a block of memory and you forget to deallocate it.
Simple question, I'm writting a program that needs to open huge image files (8kx8k) but I'm a little bit confused on how to initialize the huge arrays to hold the images in c++.
I been trying something like this:
long long SIZE = 8092*8092; ///8096*8096
double* array;
array = (double*) malloc(sizeof(double) * SIZE);
if (array == NULL)
{
fprintf(stderr,"Could not allocate that much memory");
}
But sometimes my NULL check does not catch that the array was not initialized, any idea why?
Also I can't initialize more that 2 or 3 arrays, even when running in a x64 machine with 12 GB of RAM, any idea why?
I would really wish not to have to work with sections of array instead. Any help is welcome.
Thanks.
You're not running into an array size problem. 8K*8K is merely 64M. Even 64M doubles (sizeof==8) are not an issue; that would require a mere 512 MB. Now, a 32 bit application (no matter where it's running) should be able to allocate a few of them. Not 8, because the OS typically needs to reserve some space for itself (often slightly over 2GB) and sometimes not even 3 when memory is fragmented.
The behavior of "malloc failed but didn't return NULL" is a Linux configuration bug, fixed by # echo 2 > /proc/sys/vm/overcommit_memory
malloc() does not initialize memory, it just reserves it. You will have to initialize it explicitly, e.g. via memset() from string.h:
array = (double*) malloc(SIZE * sizeof(double));
if (array) memset(array, 0, SIZE * sizeof(double));
However, in C++ you should use new instead of malloc:
double* array = new double[SIZE];
if (!array) {
cerr << "Could not allocate that much memory" << endl;
}
for (int i=0; i<SIZE; i++) array[i] = 0.0;
Regarding size: each such array is 512 MB. Are you positively sure you need double precision (which means the image has 64-bit pixel depth)? Maybe a float would suffice? That would halve the memory footprint.
You might be running into a 2GB per-process address space limit if you are running a 32bit operating system. With a few hundred MBs of system libs and other stuff, and 2 or 3 arrays of 512MB each, that will give 2GB easily. A 64bit OS would help you there.
Are you compiling your application as a 32-bit application (the default in Visual Studio, if that's what you're using), or as a 64-bit application? You shouldn't have troubles if you build it as a 64-bit app.
malloc allocates (reserves memory and returns a pointer), calloc initializes (writes all zeros to that memory).
Seems to be that you have no continuous memory block of such size (~500Mb) in C runtime heap. Instead of copying file into memory try to map image into a processes address space. You could map only necessary parts of the file.
Just as a side note: although you don't want to bother about the whole image not being in memory at once, there are reasons not to do it. Maybe think about an abstraction that allows you to keep only the currently needed chunk in memory. The program code then can be written as though ignorant of the memory issues.
I would really wish not to have to work with sections of array instead. Any help is welcome.
Have you looked into memory-mapped files?
Yep, sounds a lot like heap fragmentation, as Kirill pointed out. See also: How to avoid heap fragmentation?
i suggest using compression. decompress part of it which you need to process in your code whenever, and compress it after the part done.
2nd proposal: write code to overload memory pointer "operator+" and "operator-" so you could use non-continuous memory buffers. use smaller memory buffers make your code more stable than a continuous larger one. i had experienced it and had written some operator-overloading, see http://code.google.com/p/effoaddon/source/browse/trunk/devel/effo/codebase/addons/mem/include/mcur_i.h for the example. when i test 47G malloc()ed system memory on a x86_64, i allocated just 1G per malloc() call, so i allocated 47 memory blocks in total. EDIT: while if i tried to allocate as much as possible by using just one malloc(), i would only get 30G on a 48G system, say less than 70%, that's because larger buffer per malloc() requested, much more managemental memory consumed by the system/libc itself, you know, I called mlock() to prevent the allocated memory from being swapped out to the disk.
3rd one: try posix file mapping, map to memory per image.
Btw: call malloc() is more stable than new() though writing c++, because when memory got stressed, new() is prone to trow exceptions instead of returning NULL.
I have c++ code that attempts to dynamically allocate a 2d array of bytes that measures approx 151MB in size. When I attempt to go back and index through the array, my program crashes in exactly the same place every time with an "Access violation reading location 0x0110f000" error, but the indicies appear to be in range. That leads me to believe the memory at those indicies wasn't allocated correctly.
1) What's the max number of bytes you can dynamically allocate using the new operator?
2) If it is the case that I'm failing to dynamically allocate memory, would it make sense that my code is crashing when attempting to access the array at exactly the same two indicies every time? For some reason, I feel like they would be different every time the program is run, but what do i know ;)
3) If you don't think the problem is from an unsuccessful call to new, any other ideas what could be causing this error and crash?
Thanks in advance for all your help!
*Edit
Here's my code to allocate the 2d array...
#define HD_WIDTH 960
#define HD_HEIGHT 540
#define HD_FRAMES 100
//pHDVideo is a char**
pHDVideo->VideoData = new char* [HD_FRAMES];
for(int iFrame = 0; iFrame < HD_FRAMES; iFrame++)
{
//Create the new HD frame
pHDVideo->VideoData[iFrame] = new char[HD_WIDTH * HD_HEIGHT * 3];
memset(pHDVideo->VideoData[iFrame], 0, HD_WIDTH * HD_HEIGHT * 3);
}
and here's a screenshot of the crashing code and debugger (Dead Link) it will help.
I should add that the call to memset never fails, which to me means the allocations is successful, but I could be wrong.
EDIT
I found a fix everyone, thanks for all your help. Somehow, and I still need to figure out how, there was one extra horizontal line being upscaled, so I changed...
for(int iHeight = 0; iHeight < HD_HEIGHT; iHeight++)
to
for(int iHeight = 0; iHeight < HD_HEIGHT-1; iHeight++)
and it suddenly worked. Anyhow, thanks so much again!
Some possibilities to look at or things to try:
It may be that the pHDVideo->VideoData[iFrame] or pHDVideo->VideoData is being freed somewhere. I doubt this is the case but I'd check all the places this can happen anyway. Output a debug statement each time you free on of those AND just before your crash statement.
Something might be overwriting the pHDVideo->VideoData[iFrame] values. Print them out when allocated and just before your crash statement to see if they've changed. If 0x0110f000 isn't within the range of one of them, that's almost certainly the case.
Something might be overwriting the pHDVideo value. Print it out when allocated and just before your crash statement to see if it's changed. This depends on what else is within your pHDVideo structure.
Please show us the code that crashes, with a decent amount of context so we can check that out as well.
In answer to your specific questions:
1/ It's implementation- or platform-specific, and it doesn't matter in this case. If your new's were failing you'd get an exception or null return, not a dodgy pointer.
2/ It's not the case: see (1).
3/ See above for some possibilities and things to try.
Following addition of your screenshot:
You do realize that the error message says "Access violation reading ..."?
That means it's not complaining about writing to pHDVideo->VideoData[iFrame][3*iPixel+2] but reading from this->VideoData[iFrame][3*iPixelIndex+2].
iPixelIndex is set to 25458, so can you confirm that this->VideoData[iFrame][76376] exists? I can't see from your screenshot how this->VideoData is allocated and populated.
How are you accessing the allocated memory? Does it always die on the same statement? It looks very much like you're running off the end of either the one dimensional array of pointers, or the one of the big blocks of chars that it's pointing to. As you say, the memset pretty much proves that the memory was allocated correctly. The total amount of memory you're allocating is around 0x9450C00 bytes, so the address you quoted is off the end of allocated memory if it was allocated continguously.
Your screenshot appears to show that iPixel is in range, but it doesn't show what the value of iFrame was. Is it outside the range 0-99?
Update: The bug isn't in allocating the memory, it's in your conversion from HD to SD coordinates. The value you're reading from on the SD buffer is out of range, because it's at coordinates (144,176), which isn't in the range (0,0)-(143,175).
If it is the case that I'm failing to dynamically allocate memory, would it make sense that my code is crashing when attempting to access the array at exactly the same two indicies every time?
No, it wouldn't make sense to me.
If your call to operator new fails, I'd expect it to throw an exception or to return a null pointer (but not to return a non-null pointer to memory that's OK for some indices but not others).
Why are you using floating point math to calculate an integer index?
It looks like you are indexing out of range of the SD image buffer. 25344==SD_WIDTH*SD_HEIGHT, which is less than iPixelIndex, 25458.
Notice that heap allocation (e.g. using new) is efficient when allocating many small objects (that's why it's a Heap). If you're in the business of very large memory allocations, it might be better to use VirtualAlloc and friends.