I'm having a hard time trying to get my computer to allocate a large amount of memory (well within the 32GB on my machine). I'm in Visual Studio 2015, running the target as x64 on a 64-bit machine, so there should be no memory limitations.
Originally I tried to make vector<int> my_vector[4][180][360], which resulted in a stack overflow.
According to the Visual Studio debugger, memory usage does not go above 70MB, so I'm quite curious how the memory problems are occurring. Memory usage on the computer stays with over 25GB free.
Is there any way to declare an array vector such as vector<int> my_vector[4][180][360] without memory problems? So far I can only get as high as v[180][180]. Most of my vectors will have very few elements. Any help is much appreciated, thank you.
static std::vector<int> my_vector[4][180][360];
for (int i=0; i < 4; i++)
for (int j=0; j < 180; j++)
for (int k=0; k < 36; k++)
my_vector[i][j][k].resize(90000);
my_vector[1][2][3][4]=99;
This works on my machine with 24gb by creating virtual to disk. But it is going to be slow more likely than not. You might be better off indexing a disk file.
You can also use std::map to create a sparse array
>
static std::map<int,int> my_map[4][180][360];
my_map[1][2][3][4]=99;
Are you allocating memory on the stack? If so, I believe there is a limit before you get a stack overflow error. For Visual Studio 2015, I think the default stack size is 1MB.
For larger arrays, you need to allocate them on the heap using the new keyword.
If you are trying to allocate a multidimensional array, it can get fairly complex. A two dimensional integer array is allocated dynamically as an array of pointers, with each pointer allocated to new array of integers. In a naïve version, you will need a loop to allocate (and eventually deallocate):
int **a= new int*[1000];
for (i = 0; i< 1000; i++) {
a[i] = new int[1000];
}
As you can see, multiple dimensions become even more complex and will eat up additional memory just to store pointers. However, if you know the total number of elements, you can allocate just a single array to store all elements (100000 for my 1000x1000 example) and calculate the position of each element accordingly.
I'll leave the rest for you to figure out...
Related
I am relatively new to C++ and I've having an issue with my project.
I ran the following code a few times without a problem, but now when I try to run it it gives me a std::bad_alloc error. The code is C++ but some lines are exclusive to ROOT which is program written in C++ for particle physicists.
class Particle {
public:
int pdgid;
float px;
Particle(int pdg, float px){
pdgid = pdg;
px = px;
}
};
TFile* file = new TFile("filename.root"); //ROOT code, where particle values are obtained from.
TTree* tree = (TTree*)file->Get("FlatTree"); //tree is where all events and associated values are held
vector<Particle> allparticles;
for (unsigned iEntry = 0; iEntry<tree->GetEntries(); iEntry++) {
tree->GetEntry(iEntry);
for (int iVecEntry = 0; iVecEntry < nfsp; iVecEntry++) {
allparticles.push_back(Particle(pdg[iVecEntry],px[iVecEntry]));
}
}
The code works if I decrease the limit of the first for loop. The number of entries is quite large (over 2 million) and nfsp can be up to 24 depending on the event. This resulted in the vector allparticles having over 7 million Particle objects.
I think the problem lies with not having enough memory to allocate such a large vector but how was this working previously? Is it possible that the memory wasn't deallocated properly the first few times I ran the code?
I a bit confused about memory management. In C++ does the OS handle deallocation? Or do I have to include a destructor?
I have tried including a destructor but could not get it to work.
From "std::bad_alloc": am I using too much memory? I tried including a delete[] statement at the end of the code but this also doesn't work.
Any input and help is much appreciated!
P.S. I'm running linux mint 18.2 Sonya.
Yes, it sounds like you have run out of stack memory. Here is one of the many tutorials out there that explain heap vs stack memory.
You are creating your particles in stack memory, so this means that they will be automatically destroyed when they go out of scope. You stack memory size varies depending on the compiler and environment, but you get way less stack memory than heap memory.
To fix this, I would create a vector of pointers to Particles, and create the Particles dynamically. Example:
vector<Particle*> allparticles;
...
allparticles.push_back(new Particle(pdg[iVecEntry],px[iVecEntry]));
Remember, to delete the dynamically allocated heap memory when you are done with it. Example:
for(int i < 0; i < allparticles.size(); i++){
delete allparticles[i];
}
What I Know
I know that arrays int ary[] can be expressed in the equivalent "pointer-to" format: int* ary. However, what I would like to know is that if these two are the same, how physically are arrays stored?
I used to think that the elements are stored next to each other in the ram like so for the array ary:
int size = 5;
int* ary = new int[size];
for (int i = 0; i < size; i++) { ary[i] = i; }
This (I believe) is stored in RAM like: ...[0][1][2][3][4]...
This means we can subsequently replace ary[i] with *(ary + i) by just increment the pointers' location by the index.
The Issue
The issue comes in when I am to define a 2D array in the same way:
int width = 2, height = 2;
Vector** array2D = new Vector*[height]
for (int i = 0; i < width; i++) {
array2D[i] = new Vector[height];
for (int j = 0; j < height; j++) { array2D[i][j] = (i, j); }
}
Given the class Vector is for me to store both x, and y in a single fundamental unit: (x, y).
So how exactly would the above be stored?
It cannot logically be stored like ...[(0, 0)][(1, 0)][(0, 1)][(1, 1)]... as this would mean that the (1, 0)th element is the same as the (0, 1)th.
It cannot also be stored in a 2d array like below, as the physical RAM is a single 1d array of 8 bit numbers:
...[(0, 0)][(1, 0)]...
...[(0, 1)][(1, 1)]...
Neither can it be stored like ...[&(0, 0)][&(1, 0)][&(0, 1)][&(1, 1)]..., given &(x, y) is a pointer to the location of (x, y). This would just mean each memory location would just point to another one, and the value could not be stored anywhere.
Thank you in advanced.
What OP is struggling with a dynamically allocated array of pointers to dynamically allocated arrays. Each of these allocations is its own block of memory sitting somewhere in storage. There is no connection between them other than the logical connection established by the pointers in the outer array.
To try to visualize this say we make
int ** twodee;
twodee = new int*[4];
for (int i = 0; i < 4; i++)
{
twodee[i] = new int[4];
}
and then
int count = 1;
for (int i = 0; i < 4; i++)
{
for (int j = 0; j < 4; j++)
{
twodee[i][j] = count++;
}
}
so we should wind up with twodee looking something like
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
right?
Logically, yes. But laid out in memory twodee might look something like this batsmurph crazy mess:
You can't really predict where your memory will be, you're at the mercy of the whatever memory manager handles the allocations and what already in storage where it might have been efficient for your memory to go. This makes laying dynamically-allocated multi-dimensional arrays out in your head almost a waste of time.
And there are a whole lot of things wrong with this when you get down into the guts of what a modern CPU can do for you. The CPU has to hop around a lot, and when it's hopping, it's ability to predict and preload the cache with memory you're likely to need in the near future is compromised. This means your gigahertz computer has to sit around and wait on your megahertz RAM a lot more than it should have to.
Try to avoid this whenever possible by allocating single, contiguous blocks of memory. You may pick up a bit of extra code mapping one dimensional memory over to other dimensions, but you don't lose any CPU time. C++ will have generated all of that mapping math for you as soon as you compiled [i][j] anyway.
The short answer to your question is: It is compiler dependent.
A more helpful answer (I hope) is that you can create 2D arrays that are layed out directly in memory, or you can create "2D arrays" that are actually 1D arrays, some with data, some with pointers to arrays.
There is a convention that the compiler is happy to generate the right kind of code to dereference and/or calculate the address of an element within an array when you use brackets to access an element in the array.
Generally arrays that are known to be 2D at compile time (eg int array2D[a][b]) will be layed out in memory without extra pointers and the compiler knows to multiply AND add to get an address each time there is an access. If your compiler isn't good at optimizing out the multiply, it makes repeated accesses much slower than they can be, so in the old days we often did pointer math ourselves to avoid the multiply if possible.
There is the issue that a compiler might optimize by rounding the lower dimension size up to a power of two, so a shift can be used instead of multiply, which would then require padding the locations (then even though they are all in one memory block, there are meaningless holes).
(Also, I'm pretty sure I've run into the problem that within a procedure, it needs to know which way the 2D array really is, so you may need to declare parameters in a way that lets the compiler know how to code the procedure, eg a[][] is different from *a[]). And obviously you can actually get the pointer from the array of pointers, if that is what you want--which isn't the same thing as the array it points too, of course.
In your code, you have clearly declared a full set of the lower dimension 1D arrays (inside the loop), and you have ALSO declared another 1D array of pointers you use to get to each one without a mulitply--instead by a dereference. So all those things will be in memory. Each 1D array will surely be sequentially layed out in a contiguous block of memory. It is just that it is entirely up to the memory manager as to where those 1D arrays are, relative to each other. (I doubt a compiler is smart enough to actually do the "new" ops at compile time, but it is theoretically possible, and would obviously affect/control the behavior if it did.)
Using the extra array of pointers clearly avoids the multiply ever and always. But it takes more space, and for sequential access actually makes the accesses slower and bigger (the extra dereference) versus maintaining a single pointer and one dereference.
Even if the 1D arrays DO end up contiguous sometimes, you might break it with another thread using the same memory manager, running a "new" while your "new" inside the loop is repeating.
I have two vectors of pointers:
A vector containing pointers to all objects
A temporary vector containing pointers to some of the objects in the above vector. Each frame, this vector is cleared and pointers are then added again.
No objects are created or deleted during these frames.
According to Visual Studio's memory profiler I get a memory leak, in vector._Reallocate() every time I add a pointer to this vector.
Image: http://i.stack.imgur.com/f4Ky3.png
My question is:
Does vector.push_back() allocate something that is not deallocated later, or is Visual Studio's memory profiler just confused because I'm clearing a vector without destroying the elements? This only happens in Debug mode - not in Release.
I understand that the _Reallocate(..)-method allocates space on the heap, but since I clear the vector this should not cause a leak, I suppose?
Update: using std::list instead of std:.vector solves the "problem" - though I don't know if it really is a problem or not
Update2: It DOES happen in Release mode as well. I was a little bit confused by the output, but I get the same problem there
**Update3: I have attached some code that can reproduce the problem. I could only reproduce the problem if I store the vectors in a multidimensional array, like in the original problem.
class MyClassContainer
{
public:
std::vector<MyClass*> vec;
};
int main(int args, char **argv)
{
std::vector<MyClass*> orig;
MyClassContainer copy[101][101];
for(int i = 0; i < 101; i++)
orig.push_back(new MyClass());
while (true)
{
int rnd = std::rand() * 100 / RAND_MAX; int rnd2 = std::rand() * 100 / RAND_MAX;
for (int i = 0; i < 101; i++)
for (int j = 0; j < 101; j++)
copy[i][j].vec.clear(); // this should clear all??
copy[rnd2][rnd].vec.push_back(orig[rnd]);
}
return 0;
}
Update 4: The memory debugger shows an increasing number of heap allocations as time goes. However, I noticed now that if I wait for a long time the number of new heap allocations per second decreases towards 0. So apparently it's not really a memory leak.
It seems that when each vector in the array has been pushed to at least once nothing more gets allocated on the heap. I don't understand why though.
Not to point out the obvious... but you are not freeing the MyClass objects created here. Since you have a vector of raw pointers, they must be freed (and freed only once).
orig.push_back(new MyClass());
Put this at the end of main and your leaks should go away. Perhaps you were misled by the location of the leak report? vector::push_back() should not leak.
// Free the objects
for (int i = 0; i < orig.size(); i++) {
delete orig[i];
orig[i] = 0; // not necessary, but good defensive practice
}
I also recommend using smart-pointers, for exactly these cases.
PS: Not really my business, but what is this algorithm doing? That sure is a lot of vectors...
I want to know how many numbers can you store in array?
srand (time(NULL));
int array[10000000];
for(int i = 0; i < 10000000; i++){
array[i] = (rand() % 10000000) + 1;
}
Every time I want to store 10.000.000 numbers in array my program crashed (Eclipse). I even tryed Visual Studio and it crashed to.
So i want to know how many numbers can I store in array or is something wrong with my code?
You can store as many numbers as you have memory for, but you cannot do it like that. The reason your program crashes is that you are using an "automatic" variable, which is allocated on the "stack." The stack is much more limited in size than the "heap" typically, so using automatic variables of such large size may result in a...wait for it...
STACK OVERFLOW!
Instead, try this:
int* array = new int[10000000];
Then after using it:
delete[] array;
Step two will be to learn about smart pointers; you can use something like boost::scoped_array for this case, but there are lots of options depending on which libraries you prefer (or if you have C++11).
If you have C++11 you can use "RAII" to avoid needing to remember when and where to call delete. Just do this to allocate the array:
std::unique_ptr<int[]> array(new int[10000000]);
Or just use a vector, which always allocates its contents dynamically ("on the heap", loosely speaking):
std::vector<int> array(10000000); // 10000000 elements, all zero
The language is more than capable of storing 10,000,000 values in an array. The problem here though is you've declared the 10,000,000 elements to exist on the stack. The size of the stack is implementation dependent but most stacks simply don't have enough space for that many elements. The heap is a much better place for such an array
int* array = new int[10000000];
for(int i = 0; i < 10000000; i++){
array[i] = (rand() % 10000000) + 1;
}
...
delete[] array;
There are a couple places you could put your array in memory. The most common difference we think about is the stack and the heap.
The stack is how the computer keeps track of which function you're in, how to return from the function, and the local variables. It is often of limited size. The exact limit depends on your platform, and perhaps how you compiled your program.
The heap is another area in memory, where the compiler usually stores things you've allocated with the new keyword. This is often much larger, and capable of storing a big array such as yours. The downside to keeping things on the heap is that you have to remember to delete them at the appropriate time.
In your example, you are declaring a 10,000,000 element array on the stack. If you wanted to declare that array on the heap, you would do it like this:
srand (time(NULL));
int* array = new int[10000000];
for(int i = 0; i < 10000000; i++){
array[i] = (rand() % 10000000) + 1;
}
//Sometime later...
delete[] array;
However, C++ gives us better tools for this. If you want a big array, use std::vector.
srand (time(NULL));
std::vector<int> array(10000000);
for(std::size_t i = 0; i < array.size(); i++){
array[i] = (rand() % 10000000) + 1;
}
Now, your std::vector is on the stack, but the memory it controls is on the heap. You don't have to remember to delete it later, and your program doesn't crash.
double **array = new double* [X];
for (int i=0; i<X; i++)
array[i] = new double [Y];
array[x][y] = n;
or
double *array = new double [X*Y];
array[x*Y+y] = n;
Second version is created faster, but access is in first version faster (e.g. image processing using convolution), isn't it? Or is it all negligible?
In theory the second version should be faster, because the entire array is allocated contiguouslly, so its more cache-friendly than the first.
But in practice, profile it and see what happens. This kind of performance questions depends heavily on your architecture, OS, etc.
My advise here (In addition to profiling) is: Consider to use standard containers (A std::vector<std::vector<T>> in this case) which had been profiled, tested, and also make your life easier moving you away from raw-pointers and manual memory management.
Ok, I have: 1000x1000 image with conventional implemented Fourier transform on double arrays: Windows 7 Pro 64-bit, VC++ 2010 Express -> exactly the same (2:11 minutes)!