I am trying to compile an application but I seem to be running into a preset memory constraint. When compiling, it gives me the following error:
"virtual memory exhausted: Nicht genügend Hauptspeicher verfügbar", so I read this as having not enough RAM+Swap available.
As I am compiling this on a machine with 32GB RAM, this is quite unlikely. I checked the memory consumption and it breaks down at 3GB. Compiling the application on a different machine works, it needs around 3.5GB. I'm running on fedora 19, 64bit.
I also checked the available user memory using ulimit -a, but everything is set to unlimited (max memory size, virtual memory).
Are there any other places where there might be a limit set to the maximum memory available to a process or user? I'm starting to run out of options.
If the compiler is running out of memory, it might be due to a compiler bug, or some messed up template expansion (remember that in C++ templates are Turing complete, I remember some demented creative soul did something like computing $\pi$ to a lot of digits at compile time). Check your templates.
In case of possible compiler bug, upgrade everything. Try using clang++ instead of g++. Play with optimization and other settings.
Where does the code come from? Has somebody else built it?
Related
I'm playing around with parser combinators and have some template heavy code (lots of functions templatized over lambdas). When I try to compile it cl.exe runs up to around 660 MB of memory usage (according to Task Manager), then fails with "C1060: compiler out of heap space". I have plenty more memory available on my system (16 GB + swap file, and only half of that is being used by other processes), so I don't understand why I'm getting this error.
I have read all of the other threads about this issue and the MSDN page and tried their solutions.
Switching to the 64 bit tool architecture made the problem go away for awhile, but when my project got bigger the error returned, even though cl.exe is still nowhere near either physical or virtual memory limits.
/Zm did not have any effect whether I set it high or low, I'm not using precompiled headers anyways.
Changing /MP (multi-process compilation), whether enabling it or disabling it, had no effect.
I'm using Visual Studio Community Edition 2019 v16.11.5 and compiling with /std:c++20, if it matters.
I've pasted my code to Godbolt, though I don't know how to use multi-file projects on Godbolt so this is just to share it. If I comment out various sections of code in ExpandBrace.cpp it compiles successfully in MSVC, so I think this is an actual memory issue and is not tied to any error in the code.
I'm pretty sure my project would compile if cl.exe would use more memory before giving up, but I can't find any way to raise it's heap memory limit.
I have a program which runs for a long time, about 3 weeks. It's actually a simulation application.
After that time usually the memory gets full, the system becomes unresposive and I have to restart the whole computer. I really don't want to do that and since we are talking about Ubuntu Linux 14.04 LTS I think there is a way to avoid that. Swap is turned off, because getting stuff of the program to swap would slow it down too much.
The programm is partly written in C++ (about 10%) and FORTRAN (about 90%), and is compiled and linked using the GNU Compiler Suite (g++ and gfortran).
Getting to my question:
Is there a good way to protect the system against those programs which mess it up other than a virtual machine?
P.S.: I know the program has bugs but I cannot fix them right now, so I want to protect the system against hang ups. Also I cannot use a debugger, because it would run for too long.
Edit:
After some comments, I want to clarify some things. The code is way too complex. I don't have the time to fix the bugs and there are versions in which I don't even get the source code. I have to run it, because we are forced to do so. You do not have always the choice.
Not running a program like this is not an option because it still produces some results. So restarting the system is a workaround but I would like to do better. I consider ulimit an option, Didn't think about that one. It might help.
Limiting this crappy application memory is the easiest part. You can for example use Docker (https://goldmann.pl/blog/2014/09/11/resource-management-in-docker/#_memory), or cgroup, which are kind of virtual machine but with much less overhead. ulimit may also be an option, as mentioned in the comments.
The real problem here is to realize that if your simulation program gets killed when it runs out of memory, can you actually use the generated results? Is this program doing some checkpointing to recover from a crash?
Also badly written programs with memory leaks also frequently have more serious problems like overflows, which can turn the results totally useless if you do real science.
You may try to use valgrind to debug memory issues. Fortran also has nice compilation directives for array bounds checking, you should activate those settings if you can.
I'm compiling some code that makes a heave use of templates (its based on boost::msm framework). When compiled with g++ 4.7.1 the cc1plus process reaches about 2.4 Gb of RAM size and than fails with "virtual memory exhausted: Cannot allocate memory" error.
I'm using a 32-bit compiler (switching to 64 bit is not an option ATM), the machine itself is a 64-bit Ubuntu with 16Gb of RAM, the compilation is performed under a 64-bit chroot of Debian wheezy distribution. At the time of compilation there is plenty of RAM available, so if the compilation is to fail because of lack of physically available RAM it is to reach 4Gb first. I tried playing with "ulimit -m" options, setting to different values and setting it to smaller sizes causes the compiler to fail earlier but when left to "unlimited" it fails at the above mentions 2+ Gb.
So I guess something else must be limiting me. Maybe someone encountered a similar issue and knows a way to change the limitation?
In 32-bit application (including compilers), you typically get somewhere between 2 and 3GB that is available for usermode in virtual space. This is caused by a combination of memory space being reserved, memory space fragmentation (there is virtual memory available, just not a big enough chunk to hold whatever size block that new or malloc is requesting), and "memory reservation", where process has allocated a fairly large chunk of memory, but it's not actually using all of it, so it's not "populated".
Any particular reason you can't use a 64-bit GCC to generate 32-bit code - using -M32? That would be my solution.
I have a c++ program which I'm running on a windows 7 64bit machine, using Eclipse as my IDE. I use mingw32 for a compiler.
The problem: When I debug the program using the gdb debugger it runs just fine and does what it needs to do. But when I run it without debugging, either from the command line or from within eclipse (using the same configuration as with the debug), it crashes.
I tried running the program from the command line and attaching to the process using the debugger, and what I saw is that it reaches the following line of code:
anc_map[ancestry].hap_array = (char**)calloc(anc_map[ancestry].nr_hap , sizeof(char*));
and just hangs (cpu is not working, and nothing happens although the program is still running).
The above line is actually called more than once, and the hanging occurs the second time it is called (it works the first time).
Any idea what can be the cause for this behavior?
Thanks,
Itamar.
Edit:
I realize that using calloc is a little old-fashioned, but since this is a legacy code I just need to modify a little, I'm trying to avoid doing major refactoring.
I've tried compiling the code and running on linux, and the problem does not occur there, so it has something to do with my configuration on the windows machine
First thing that comes to mind is whether anc_map[ancestry].nr_hap could be some bogus, probably huge, number. Probably because any of the variables got corrupt. I am not sure why it would get corrupt only without debugger, but it might be that the debugger affects where things are allocated and the corruption appears somewhere less harmful when debugging.
The other thing that comes to mind is, that if the program needs a lot of memory, the debugger might affect the 2-GB limit flag in Windows, so in one case there is enough memory and the other way you run out. I am, however, not sure how to change it with mingw32 compiler, as I only did it with the Microsoft one (/LARGEADDRESSAWARE option to Microsoft link and editbin). The reason is, that in some old software, they noticed people doing binary search like (whatever *)(((unsigned)begin + (unsigned)end)/2), which, besides being incorrect C, does not work if the pointers are above 2GB, because the calculation overflows. So for old software, written before more than 2GB was common, they limited the memory to 2GB and provided option to get more, which means 3GB on 32-bit Windows (the last 1GB maps kernel space to avoid swapping page tables on kernel entry and exit; linux does the same thing) and 4GB for 32-bit process on 64-bit windows (the kernel can be mapped above 4GB there).
Hm, but most likely it's actually corruption of the memory management metadata, because that's the usual case where memory allocation or deallocation functions just hang instead of returning an error. Again the debugger would cause some addresses to be different and the corruption to happen elsewhere.
In the first and last case the corruption would probably be always there, so you might have some luck trying to run it:
In linux under valgrind.
Under DUMA, but the Microsoft standard runtime library will try to resist replacing the memory allocation functions rather hard; I finally gave up when I found that IO streams use something like __debug_delete, but normal new. Or the other way around; I don't recall exactly.In either case one was the standard allocation function and the other was some their internal undocumented function. It will also use much more memory than usual, because each allocation will be at least 8kB. In Linux it's trivial, because GNU libc has special support for overriding memory allocation, but valgrind is superior there anyway.
I have the following problem:
A program run on a windows machine (32bit, 3.1Gb memory, both VC++2008 and mingw compiled code) fails with a bad_alloc exception thrown (after allocating around 1.2Gb; the exception is thrown when trying to allocate a vector of 9 million doubles, i.e. around 75Mb) with plenty of RAM still available (at least according to task manager).
The same program run on linux machines (32bit, 4Gb memory; 32bit, 2Gb memory) runs fine with peak memory usage of around 1.6Gb. Interestingly the win32 code generated by mingw run on the 4Gb linux machine under wine also fails with a bad_alloc, albeit at a different (later) place then when run under windows...
What are the possible problems?
Heap fragmentation? (How would I know? How can this be solved?)
Heap corruption? (I have run the code with pageheap.exe enabled with no errors reported; implemented vector access with bounds checking --- again no errors; the code is essentially free of pointers, only std::vectors and std::lists are used. Running
the program under Valgrind (memcheck) consumes too much memory and ends prematurely, but does not find any errors)
Out of memory??? (There should be enough memory)
Moreover, what could be the reason that the windows version fails while the
linux version works (and even on machines with less memory)? (Also note that
the /LARGEADDRESSAWARE linker flag is used with VC+2008 if that can have any effect)
Any ideas would be much appreciated, I am at my wits end with this... :-(
It has nothing to do with how much RAM is in your system. You are running out of virtual address space. For a 32 bit windows OS process, you get a 4GB virtual address space (irrespective of how much RAM you are using) out of 2GB for the user-mode (3GB in case of LARGEADDRESSAWARE) and 2 GB for kernel. When you do try to allocate memory using new, OS will try to find the contiguos block of virtual memory which is large enough to satisfy the memory allocation request. If your virtual address space is badly fragmented or you are asking for a huge block of memory then it will fail throwing a bad_alloc exception. Check how much virtual memory your process is using.
With Windows XP x86 and the default settings, 1.2 GB is about all the address space you have left for your heap after system libraries, your code, the stack and other stuff get their share. Note that largeaddressaware requires you to boot with the /3GB boot flag to try to give your process up to 3GB. The /3GB flag causes instability on a lot of XP systems, which is why it's not enabled by default.
Server variants of Windows x86 give you more address space, both by using the 3GB/1GB split and by using PAE to allow the use of your full 4GB of RAM.
Linux x86 uses a 3GB/1GB split by default.
A 64 bit OS would give you more address space, even for a 32bit process.
Are you compiling in Debug mode? If so, the allocation will generate a huge amount of debugging data which might generate the error you have seen, with a genuine out-of-memory. Try in Release to see if that solves the problem.
I have only experienced this with VC, not MinGW, but then I haven't checked either, this could still explain the problem.
To elaborate more about the virtual memory:
Your application fails when it tries to allocate a single 90MB array, and there is no contiguous space of virtual memory where this can fit left. You might be able to get a little farther if you switched to data structures that use less memory -- perhaps some class that approximates a huge array by using a tree where all data is kept in 1MB (or so) leaf nodes. Also, under c++ when doing a huge amount of allocations, it really helps if all those big allocations are of same size, this helps reusing memory and keeps fragmentation much lower.
However, the correct thing to do in the long run is simply to switch to a 64-bit system.