I wrote a program in c++ to perform montecarlo. The thing is that after five iterations (every iteration runs monte carlo with different configurations), the process is killed.
At the begining I thought it was a memory problem but after reading this nice post on memory management (http://stackoverflow.com/questions/76796/memory-management-in-c), my scoping seems to be correct.
I do not use a lot of memory since my results are stored in a relativelly small array which is rewritten ever iteration. In an iteration I am not using more memory than in the previous.
I cannot find, if there is any, where is the leak. I have a lot of function calls to perform
the calculations, but I do not need to destroy the objects once I am out of the function right?
Any tip?
EDIT: The program takes all the processor power of my computer, when it is running I cannot even move the mouse.
Thanks in advance.
EDIT SOLVED: The problem was that I was not deletening the pointers I used so every iteration the memory was not delocated and a whole new set of pointers was created using more memory. Thanks a lot to those that answered.
Depending on the platform you are on, you can use a tools like valgrind or vld to find memory leaks in your program.
Related
How does one best identify memory not released properly during run time? I know of several programs that identifies allocated and non freed (leaked) memory when the application closes. But my issue seems to be that during the program execution (possibly a thread) creates some objects that are not freed, although they should be after the system is done with the "work".
Keeping the system running this builds up over time. But when the program shuts down the memory seems to be freed correctly and thus never reported as a leak in MadExcept that I use at the moment.
How do I best go about to detect what is allocating this memory every time the "work" is run and not freeing it until program termination? This is in quite a large server system with around 1 million lines of code, several DLL sub projects and multiple threads running (40-50).
Perhaps there is some system that could identify allocated objects that have been alive for longer than X min. Let's say 60 min is selected and the system left running. Then this information could be used to locate many of these long living objects and investigate those?
if you are using c++ and visual studio, I think this link is helpful. You can _CrtMemCheckpoint and CrtMemDumpStatistics when you need.
I ended up trying the evaluation version of Softwareverify's C++ Memory Validator.
It worked just like what I wanted and was able to provide a time line of memory allocations etc to allow me to identify what had been accumulating over time and how long it had been alive. Using that I was able to identify the problem and fix it.
I have two threads running in a program.
They are created using boost::thread.
The two threads do not share anything in terms of memory. No data-structures or objects are shared between them.
Now the second thread uses a class which has as private members a lot of Eigen double Matrices.
I make sure that the matrices are aligned using the Eigen directive EIGEN_MAKE_ALIGNED_OPERATOR_NEW etc
When the first thread is running the elements at the matrices of the second class are over-written .
I checked that by inspections since elements that should be decimals suddenly become integers.
When the first thread is not running the second has no problem and its Eigen members have correct values.
Again:
1) The two threads share no data structures.
2) There is no segmentation fault message or something similar or some error message while the program is running.
3) Any suggestions how to protect the memory of the second thread or how to track done how the memory is violated?
Thank you in advance. I am really sorry I did not post code but it is huge.
Let me know if you want me to post something specific from the code.
You likely want a debugging tool like mallocguard for Mac or Electric Fence for Linux.
These work by adding "Guard Pages" before allocations which mark them as inaccessible virtual memory. When the memory is freed, it too is marked as inaccessibile. If the program attempts to access memory that it shouldn't, the modified allocator ensures that it crashes immediately, so that your debugger will hopefully highlight the line of code that was causing the corruption. Beware that this can consume large amounts of memory, so you'll potentially need a small data set that reproduces the corruption.
I am developing a "pyramid" game that uses a Minimax tree that searches for the "best" move .. but my game is freezing.
My deduction is that it's a memory problem but I am using only 124kb of memory. How much memory can I allocate with the new operator? Or, what is the memory limit assigned to my application by default.
The OS does decide how much memory you'll be able to allocate at each given time, making your question impossible to answer.
I would be you, I would show the faulty code instead of thinking it's the compiler or the environnement that does something wrong. 124kb is nothing on most platforms.
If it's not throwing a bad_alloc, I really doubt your new call will fail. It's certainly not the real problem.
You might also consider looking at possible stack overflow. Especially if your algorithm involves recursion.
There is no limit by default.
If you allocate smallish objects, you can generally get 1 to 1.5 GB of them on 32-bit Windows.
I don't think a memory problem would cause your game to freeze before it would cause it to crash. You probably have an infinite loop somewhere. If you had a problem with the memory, I would imagine you would get an error message of some sort as opposed to the loop issue where it will just hang.
It's probably not you are consuming all of memory, but you are consuming all of CPU!
Check the logic of your game program.
I'm building a large RTree (spatial index) full of nodes. It needs to be able to handle many queries AND updates. Objects are continuously being created and destroyed. The basic test I'm running is to see the performance of the tree as the number of objects in the tree increases. I insert from 100-20000 uniformly size, randomly located objects in increments of 100. Searching and updating are irrelevant to the issue I am currently faced with.
Now, when there is NO memory leak the "insert into tree" performance is everywhere. It goes anywhere from 10.5 seconds with ~15000 objects to 1.5 with ~18000. There is no pattern whatsoever.
When I deliberately add in a leak, as simple as putting in "new int;" I don't assign it to anything, that right there is a line to itself, the performance instantly falls onto a nice gentle curve sloping from 0 (roughly) seconds for 100 objects to 1.5 for the full 20k.
Very, very lost at this point. If you want source code I can include it but it's huuugggeeee and literally the only line that makes a difference is "new int;"
Thanks in advance!
-nick
I'm not sure how you came up with this new int test, but it's not very good way to fix things :) Run your code using a profiler and find out where the real delays are. Then concentrate on fixing the hot spots.
g++ has it built in - just compile with -pg
Without more information it's impossible to be sure.
However I wonder if this is to do with heap fragmentation. By creating a freeing many blocks of memory you'll likely be creating a whole load of small fragments of memory linked together.The memory manager needs to keep track of them all so it can allocate them again if needed.
Some memory managers when you free a block try to "merge" it with surrounding blocks of memory and on a highly fragmented heap this can be very slow as it tries to find the surrounding blocks. Not only this, but if you have limited physical memory it can "touch " many physical pages of memory as it follows the chain of memory blocks which can cause a whole load of extremely slow page faults which will be very variable in speed depending on exactly how much physical memory the OS decides to give that process.
By leaving some un-freed memory you will be changing this pattern of access which might make a large difference to the speed. You might for example be forcing the run time library to allocate new block of memory each time rather than having to track down a suitably sized existing block to reuse.
I have no evidence this is the case in your program, but I do know that memory fragmentation is often the causes of slow programs when a lot of new and free is performed.
The possible thing that is happening which explains this (a theory)
The compiler did not remove the empty new int
The new int is in one of the inner loops or somewhere in your recursive traversal wherein it gets executed the most amount of time
The overall RSS of the process increases and eventually the total memory being used by the process
There are page faults happening because of this
Because of the page-faults, the process becomes I/O bound instead of being CPU bound
End result, you see a drop in the throughput. It will help if you can mention the compiler being used and the options for the compiler that you are using to build the code.
I am taking a stab in the dark here but the problem could be the way the heap gets fragmented. You said that you are creating a destroying large numbers of objects. I will assume that the objects are all of different size.
When one allocates memory on the heap, a cell the size needed is broken off from the heap. When the memory is freed, the cell is added to a freelist. When one does a new alloc, the allocator walks the heap until a cell that is big enough is found. When doing large numbers of allocations, the free list can get rather long and walking the list can take a non-trivial amount of time.
Now an int is rather small. So when you do your new int, it may well eat up all the small heap cells on the free list and thus dramatically speed up larger allocations.
The chances are, however that you are allocating and freeing similar sized objects. If you use your own freelists, you will safe yourself many heap walks and may dramatically improve performance. This is exactly what the STL allocators do to improve performance.
Solution: Do not run from Visual Studio. Actually run the .exe file. Figured this out because that's what the profilers were doing and the numbers were magically dropping. Checked memory usage and version running (and giving me EXCEPTIONAL times) was not blowing up to excessively huge sizes.
Solution to why the hell Visual Studio does ridiculous crap like this: No clue.
I have the need to allocate large blocks of memory with new.
I am stuck with using new because I am writing a mock for the producer side of a two part application. The actual producer code is allocating these large blocks and my code has responsibility to delete them (after processing them).
Is there a way I can ensure my application is capable of allocating such a large amount of memory from the heap? Can I set the heap to a larger size?
My case is 64 blocks of 288000 bytes. Sometimes I am getting 12 to allocate and other times I am getting 27 to allocate. I am getting a std::bad_alloc exception.
This is: C++, GCC on Linux (32bit).
With respect to new in C++/GCC/Linux(32bit)...
It's been a while, and it's implementation dependent, but I believe new will, behind the scenes, invoke malloc(). Malloc(), unless you ask for something exceeding the address space of the process, or outside of specified (ulimit/getrusage) limits, won't fail. Even when your system doesn't have enough RAM+SWAP. For example: malloc(1gig) on a system with 256Meg of RAM + 0 SWAP will, I believe, succeed.
However, when you go use that memory, the kernel supplies the pages through a lazy-allocation mechanism. At that point, when you first read or write to that memory, if the kernel cannot allocate memory pages to your process, it kills your process.
This can be a problem on a shared computer, when your colleague has a slow core leak. Especially when he starts knocking out system processes.
So the fact that you are seeing std::bad_alloc exceptions is "interesting".
Now new will run the constructor on the allocated memory, touching all those memory pages before it returns. Depending on implementation, it might be trapping the out-of-memory signal.
Have you tried this with plain o'l malloc?
Have you tried running the "free" program? Do you have enough memory available?
As others have suggested, have you checked limit/ulimit/getrusage() for hard & soft constraints?
What does your code look like, exactly? I'm guessing new ClassFoo [ N ]. Or perhaps new char [ N ].
What is sizeof(ClassFoo)? What is N?
Allocating 64*288000 (17.58Meg) should be trivial for most modern machines... Are you running on an embedded system or something otherwise special?
Alternatively, are you linking with a custom new allocator? Does your class have its own new allocator?
Does your data structure (class) allocate other objects as part of its constructor?
Has someone tampered with your libraries? Do you have multiple compilers installed? Are you using the wrong include or library paths?
Are you linking against stale object files? Do you simply need to recompile your all your source files?
Can you create a trivial test program? Just a couple lines of code that reproduces the bug? Or is your problem elsewhere, and only showing up here?
--
For what it's worth, I've allocated over 2gig data blocks with new in 32bit linux under g++. Your problem lies elsewhere.
It's possible that you are being limited by the process' ulimit; run ulimit -a and check the virutal memory and data seg size limits. Other than that, can you post your allocation code so we can see what's actually going on?
Update:
I have since fixed an array indexing bug and it is allocating properly now.
If I had to guess... I was walking all over my heap and was messing with the malloc's data structures. (??)
i would suggest allocating all your memory at program startup and using placement new to position your buffers. why this approach? well, you can manually keep track of fragmentation and such. there is no portable way of determining how much memory is able to be allocated for your process. i'm positive there's a linux specific system call that will get you that info (can't think of what it is). good luck.
The fact that you're getting different behavior when you run the program at different times makes me think that the allocation code isn't the real problem. Instead, somebody else is using the memory and you're the canary finding out it's missing.
If that "somebody else" is in your program, you should be able to find it by using Valgrind.
If that somebody else is another program, you should be able to determine that by going to a different runlevel (although you won't necessarily know the culprit).