int **something = new int *[N];
for(int n = 0; n < N; n++)
something[n] = new int[M];
#pragma omp parallel for
for (int n = 0; n < N; n++)
delete[] something[n];
delete[] something;
Can I parallelize a delete process like this?
OOM killer of Linux killed my process after quite big number of loops. I tried to figure out where the memory is leaking, but I could not find where. I am not sure if these for loop delete process worked well or not.
This is generally okay. The standard (g)libc heap must be thread-safe; otherwise, it'd be impossible to write threaded programs at all. It's also fine to allocate a buffer on one thread and free it on another.
Parallelizing heap allocations won't give you much of a speed-up, but I suspect you're asking not because you want to speed up (de-)allocation, but rather because you have some other parallel code that happens to allocate and free some buffers.
To find your leak, try running your program with valgrind.
Additionally, consider calling malloc_trim(0); after freeing large amounts of memory. This makes glibc release unused memory back to the operating system, instead of holding onto all of it for later allocations.
Related
In C++, the importance of deallocating memory when the program is either exiting or no longer serves a purpose is important. So if this is allocation of a dynamic array
char** dynamicArr = nullptr;
for (int i = 0; i < x; i++) {
mapPtr[i] = new char[y];
}
and this is deallocation of a dynamic array
for (int i = 0; i < x; i++) {
delete[] mapPtr[i];
}
delete[] mapPtr;
However, when it comes to vectors, I noticed that my global vector with 0 elements inside seems to be causing some memory leaks.
I've read up on this link, a user commented that
No. The std::vector will automatically de-allocate the memory it uses
Screenshot of my debugging.
I have also tried these steps to clear the vector as well as make sure the vector inside the struct citySummInfo has shrunk to fit and clear hopefully not getting any memory leak but to no avail. Is there any way that I'm doing it wrong?
As what #JohnFilleau have mentioned
_CrtDumpMemoryLeaks() should be called at the point in the program where you want to see what is remaining on the heap. Since your
vectors are statically allocated, they will not have been destroyed at
the time you call this function.
_CrtDumpMemoryLeaks() is meant to place right before the program terminates, and since my vectors are statically allocated, it has not been deallocated at the time when _CrtDumpMemoryLeaks() has been called hence the "leaks".
As I know STL has automatic memory management. But when I use something like top or ps -aux to show the memory usage of a process, it shows even the STL object are out of scope, these memory is still possessed by process.
Here is an example:
void run()
{
map<int, int> a;
for(int i = 0; i < 1000000; i++)
{
a[i] = i;
} // 64376K memory used by process
}
int main()
{
run();
sleep(5); // still 64376 memory used
map<int, int> a;
for(int i = 0; i < 1000000; i++)
{
a[i] = i;
} // still 64376 memory used
return 0;
}
The process possesses 64376KB memory in run() and memory doesn't release after function run(). But these memory seems to be used by the second map.
After I use valgrind --tool=massif to check what happened, I got a normal result.
So here comes my question
why process memory trends doesn't match with the code and valgrind
How does the different STL objects share the same allocated memory.
This is completely normal. That's how operating systems work. If they spent all their time reclaiming tiny portions of memory from tiny processes, they'd never do anything else.
You just have to trust that their complicated algorithms know what they're doing to get the best performance for your system.
There are layers on layers on layers of logic that allocate physical RAM all the way up to the process's virtual memory.
Going into extreme detail about how operating systems work would be both beyond the scope of this post, and pointless. However, you could enrol on a relevant teaching course if you really wanted to grok it all.
If we were to forget about caching and virtual memory and such, even a simple rule of thumb might be summarised as follows: releasing memory from your program tells the OS it can have it back; that doesn't mean the OS must take it back.
I am relatively new to C++ and I've having an issue with my project.
I ran the following code a few times without a problem, but now when I try to run it it gives me a std::bad_alloc error. The code is C++ but some lines are exclusive to ROOT which is program written in C++ for particle physicists.
class Particle {
public:
int pdgid;
float px;
Particle(int pdg, float px){
pdgid = pdg;
px = px;
}
};
TFile* file = new TFile("filename.root"); //ROOT code, where particle values are obtained from.
TTree* tree = (TTree*)file->Get("FlatTree"); //tree is where all events and associated values are held
vector<Particle> allparticles;
for (unsigned iEntry = 0; iEntry<tree->GetEntries(); iEntry++) {
tree->GetEntry(iEntry);
for (int iVecEntry = 0; iVecEntry < nfsp; iVecEntry++) {
allparticles.push_back(Particle(pdg[iVecEntry],px[iVecEntry]));
}
}
The code works if I decrease the limit of the first for loop. The number of entries is quite large (over 2 million) and nfsp can be up to 24 depending on the event. This resulted in the vector allparticles having over 7 million Particle objects.
I think the problem lies with not having enough memory to allocate such a large vector but how was this working previously? Is it possible that the memory wasn't deallocated properly the first few times I ran the code?
I a bit confused about memory management. In C++ does the OS handle deallocation? Or do I have to include a destructor?
I have tried including a destructor but could not get it to work.
From "std::bad_alloc": am I using too much memory? I tried including a delete[] statement at the end of the code but this also doesn't work.
Any input and help is much appreciated!
P.S. I'm running linux mint 18.2 Sonya.
Yes, it sounds like you have run out of stack memory. Here is one of the many tutorials out there that explain heap vs stack memory.
You are creating your particles in stack memory, so this means that they will be automatically destroyed when they go out of scope. You stack memory size varies depending on the compiler and environment, but you get way less stack memory than heap memory.
To fix this, I would create a vector of pointers to Particles, and create the Particles dynamically. Example:
vector<Particle*> allparticles;
...
allparticles.push_back(new Particle(pdg[iVecEntry],px[iVecEntry]));
Remember, to delete the dynamically allocated heap memory when you are done with it. Example:
for(int i < 0; i < allparticles.size(); i++){
delete allparticles[i];
}
I'm having a hard time trying to get my computer to allocate a large amount of memory (well within the 32GB on my machine). I'm in Visual Studio 2015, running the target as x64 on a 64-bit machine, so there should be no memory limitations.
Originally I tried to make vector<int> my_vector[4][180][360], which resulted in a stack overflow.
According to the Visual Studio debugger, memory usage does not go above 70MB, so I'm quite curious how the memory problems are occurring. Memory usage on the computer stays with over 25GB free.
Is there any way to declare an array vector such as vector<int> my_vector[4][180][360] without memory problems? So far I can only get as high as v[180][180]. Most of my vectors will have very few elements. Any help is much appreciated, thank you.
static std::vector<int> my_vector[4][180][360];
for (int i=0; i < 4; i++)
for (int j=0; j < 180; j++)
for (int k=0; k < 36; k++)
my_vector[i][j][k].resize(90000);
my_vector[1][2][3][4]=99;
This works on my machine with 24gb by creating virtual to disk. But it is going to be slow more likely than not. You might be better off indexing a disk file.
You can also use std::map to create a sparse array
>
static std::map<int,int> my_map[4][180][360];
my_map[1][2][3][4]=99;
Are you allocating memory on the stack? If so, I believe there is a limit before you get a stack overflow error. For Visual Studio 2015, I think the default stack size is 1MB.
For larger arrays, you need to allocate them on the heap using the new keyword.
If you are trying to allocate a multidimensional array, it can get fairly complex. A two dimensional integer array is allocated dynamically as an array of pointers, with each pointer allocated to new array of integers. In a naïve version, you will need a loop to allocate (and eventually deallocate):
int **a= new int*[1000];
for (i = 0; i< 1000; i++) {
a[i] = new int[1000];
}
As you can see, multiple dimensions become even more complex and will eat up additional memory just to store pointers. However, if you know the total number of elements, you can allocate just a single array to store all elements (100000 for my 1000x1000 example) and calculate the position of each element accordingly.
I'll leave the rest for you to figure out...
I have two vectors of pointers:
A vector containing pointers to all objects
A temporary vector containing pointers to some of the objects in the above vector. Each frame, this vector is cleared and pointers are then added again.
No objects are created or deleted during these frames.
According to Visual Studio's memory profiler I get a memory leak, in vector._Reallocate() every time I add a pointer to this vector.
Image: http://i.stack.imgur.com/f4Ky3.png
My question is:
Does vector.push_back() allocate something that is not deallocated later, or is Visual Studio's memory profiler just confused because I'm clearing a vector without destroying the elements? This only happens in Debug mode - not in Release.
I understand that the _Reallocate(..)-method allocates space on the heap, but since I clear the vector this should not cause a leak, I suppose?
Update: using std::list instead of std:.vector solves the "problem" - though I don't know if it really is a problem or not
Update2: It DOES happen in Release mode as well. I was a little bit confused by the output, but I get the same problem there
**Update3: I have attached some code that can reproduce the problem. I could only reproduce the problem if I store the vectors in a multidimensional array, like in the original problem.
class MyClassContainer
{
public:
std::vector<MyClass*> vec;
};
int main(int args, char **argv)
{
std::vector<MyClass*> orig;
MyClassContainer copy[101][101];
for(int i = 0; i < 101; i++)
orig.push_back(new MyClass());
while (true)
{
int rnd = std::rand() * 100 / RAND_MAX; int rnd2 = std::rand() * 100 / RAND_MAX;
for (int i = 0; i < 101; i++)
for (int j = 0; j < 101; j++)
copy[i][j].vec.clear(); // this should clear all??
copy[rnd2][rnd].vec.push_back(orig[rnd]);
}
return 0;
}
Update 4: The memory debugger shows an increasing number of heap allocations as time goes. However, I noticed now that if I wait for a long time the number of new heap allocations per second decreases towards 0. So apparently it's not really a memory leak.
It seems that when each vector in the array has been pushed to at least once nothing more gets allocated on the heap. I don't understand why though.
Not to point out the obvious... but you are not freeing the MyClass objects created here. Since you have a vector of raw pointers, they must be freed (and freed only once).
orig.push_back(new MyClass());
Put this at the end of main and your leaks should go away. Perhaps you were misled by the location of the leak report? vector::push_back() should not leak.
// Free the objects
for (int i = 0; i < orig.size(); i++) {
delete orig[i];
orig[i] = 0; // not necessary, but good defensive practice
}
I also recommend using smart-pointers, for exactly these cases.
PS: Not really my business, but what is this algorithm doing? That sure is a lot of vectors...