When assigning values to a large array the used memory keeps increasing even though no new memory is allocated. I am checking the used memory simply by the task manager (windows) or system monitor (Ubuntu).
The Problem is the same on both OS. I am using gcc 4.7 or 4.6 respectively.
This is my code:
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char *argv[]) {
int i,j;
int n=40000000; //array size
int s=100;
double *array;
array=malloc(n*sizeof(double)); //allocate array
if(array==NULL){
return -1;
}
for(i=0;i<n;i++){ //loop for array, memory increases during this loop
for(j=0;j<s;j++){ //loop to slow down the program
array[i] = 3.0;
}
}
return 0;
}
I do not see any logical Problem, but to my knowledge I do not exceed any system limits either. So my questions are:
can the problem be reproduced by others?
what is the reason for the growing memory?
how do I solve this issue?
When modern systems 'allocate' memory, the pages are not actually allocated within physical RAM. You will get a virtual memory allocation. As you write to those pages, a physical page will be taken. So the virtual RAM taken will be increased when you do the malloc(), but only when you write the value in will the physical RAM be taken (on a page by page basis).
You should see the virtual memory used increase immediately. After that the RSS, or real memory used will increment as you write into the newly allocated memory. More information at How to measure actual memory usage of an application or process?
This is because memory allocated in Linux and on many other operating systems, isn't actually given to your program until you use it.
So you could malloc 1 GB on a 256 MB machine, and not run out of memory until you actually tried to use all 1 GB.
In Linux there is a group of overcommit settings which changes this behavior. See Cent OS: How do I turn off or reduce memory overcommitment, and is it safe to do it?
Related
If I need a program to r/w data larger then 1T randomly, a simplest way is putting all data into memory. For the PC with 2G memory, we can nevertheless work by doing a lot of i/o. Different computers have different size memory, so how to alloc suitable memory on computers from 2G to 2T memory in one program?
I thought of popen /proc/meminfo and alloc MemFree but I think it maybe have a better way.
note:
use Linux, but other OS is welcome
avoid being OOM killed as well as possible (without root)
disk i/o as less as possible
use multiprocessing
c or c++ answer is fine
You can use the GNU extension get_avphys_pages() from glibc
The get_phys_pages function returns the number of available pages of physical the system has. To get the amount of memory this number has to be multiplied by the page size.
Sample code:
#include <unistd.h>
#include <sys/sysinfo.h>
#include <stdio.h>
int main() {
long int pagesize = getpagesize();
long int avail_pages = get_avphys_pages();
long int avail_bytes = avail_pages * pagesize;
printf( "Page size:%ld Pages:%ld Bytes:%ld\n",
pagesize, avail_pages, avail_bytes );
return 0;
}
Result Godbolt
Program returned: 0
Page size:4096 Pages:39321 Bytes:161058816
This is the amount of PHYSICAL memory in your box so:
The true available memory can be much higher if the process pages in/out
The physical memory is a maximum as there would be other processes using memory too.
So treat that result as an estimate upper bound for available DDR.
If you plan to allocate large chunks of memory use mmap() directly as malloc() would be too high level for this usage.
I created a simple program that allocates a integer array onto the heap/free store every time the user clicks 'enter'. My ulitmate goal was to see a change in the heap memory size with each new allocation. I do this by checking the status file to see the change in heap memory size. Unfortunately, at each allocation, the status file doesn't update. It seems like the only way I could get the file to update is if I spam the program with a bunch of carriage returns and then it updates. Why does it do this? I need a reliable method to determine the range of memory my programs are taking up from dynamic allocations as I am working on an embedded system. Can anyone provide any insights to /proc//status or another method one could determine heap memory size?
Also status provides the memory in KB, it would be nice to see this information with more granularity to the byte - is there a way to do this? Thank you.
Here is a sample program of what I did:
#include <cstddef>
#include <iostream>
int keepLooping()
{
return 1;
}
int main (int argc, char* argv[])
{
int exit_code = 0;
int fd = -1;
do
{
do
{
std::cout << '\n' << "Press a key to allocate data to the heap...";
} while (std::cin.get() != '\n');
int *someArray = new int[1000];
}
while (keepLooping());
return exit_code;
}
C++ implementations don't typically allocate every dynamic allocation directly from the operating system, but rather request entire pages of memory (4kB by default on Linux) or more and distribute smaller allocations within the process. Therefore it is unlikely to be able for the operating system to observe smaller changes in the memory use of the process.
On some systems, Linux in particular given certain configuration, allocating dynamic storage doesn't necessarily use any system resources. Such systems which over commit the memory do not see any change in the use of physical memory until the allocated memory is actually accessed.
There are memory profiling tools such as valgrind which can be used to wrap the process and accurately measure the use of heap memory throughout the execution of the process.
I need to acquire several GB of data from a sensor. When I tried to allocate a big array with malloc (10 or more GB. My system has 32GB) it returns NULL. So I thought the problem could be solved with a linked list of iterators to vectors.
However I don't know how to set this up. I tried declaring "
list< vector::iterator >" but I can't allocate the memory for each vector (e/o should have 1000~2000 elements). Do you know any way to do this or maybe a better solution for this big memory allocation?
If you are using a 64-bit operating system, then malloc should be able to allocate the large size with no problem.
For example, this code runs on my windows machine (64-bit windows) and allocates 10GB of ram flawlessly:
#include <stdio.h>
#include <malloc.h>
int main(int argc, char *argv[]) {
long int size = 10L * 1024 * 1024 * 1024;
printf("size = %ld\n", size);
char *x = (char *)malloc(size);
printf("x = 0x%lx\n", x);
long int i;
for (i = 0; i < size; i += 1024*1024) {
x[i] = 'h';
}
printf("Done1\n");
}
However, if you have a 32-bit operating system, you'll be in trouble, and can't allocate over some limit (maybe 3 GB, but probably system dependent)
In that case, you'll need to write your data to a file instead.
However, if you're using a fat filesystem, then you can't write to a file that big either. In that case, you'd have to split the data among many files under 2gb in size.
You'd want to actually check the malloc result for NULL to make sure the malloc works and memory could be grabbed.
You will need to allocation this space under Windows 64 bit OS. You will ALSO have to set "large address space aware" flag, otherwise you can only get 2 GB of RAM due to how the virtual memory system works on Windows.
You may want to look into using a memory mapped file, as suggested by sehe in his answer if you do not absolutely have to have one large 10 GB chunk of continuous memory. If you have to build your application for Windows 32 bit, then this will be the only answer, as Windows 32 bit normally only allows for 2 GB of memory, unless the option is set for "large address space aware" flag, at which point it will allow 3 GB of memory usage.
When you have to deal with large blocks of memory, you are better off skipping malloc altogether and going directly to the operating system calls for memory allocation.
I usually move to memory mapped files or shared memory maps for this kind of data volumes.
This way, you're not bound to the amount of physical (process) memory available at all. You can let the OS page in and out as required. Fragmentation becomes much less of an issue (unless you actually fragment the logical address space, which is quite hard to achieve on 64 bit architectures).
More information
I have quite a number of answers on SO that show examples of storing vectors and all manner of more complicated data structures in shared memory/mapped files. You might want to look for mapped_file_device (from Boost Iostreams) or managed_shared_memory and managed_mapped_file (from Boost Interprocess)
I am interested to find out what is the memory limit for the automatic and dynamically-allocated variables, so I tested things like that:
int main() {
const int N = 1000000;
int a[N];
a[1] = 100;
}
I found the maximum N that would not incur a Segmentation fault is 2,600,000, about 10MB.
Then I tested dynamically allocated variables, like this:
int main() {
const int N = 1000000;
int* a = new int [N];
delete[] a;
}
I found that maximum N that would not throw an exception is about 730,000,000, that's about 3GB.
Now the question is, how is the 10MB limit (for automatic variables) and 3GB limit (for dynamically-allocated variables) determined. I assume it is related to my machine? Also, is there any way to increase the limit, in case I really need it?
Language mandates nothing. It's all implementation-defined.
Automatic variables usually go onto stack, and you can usually increase the maximum size via compiler options. Free store is usually heap, and is limited only by usable address space. Don't count on more than 2-3GB in 32-bit environment, the limit will be much higher in 64-bit environment. Of course, you won't be able to allocate all of the 64-bit address space, you'll hit the limit of available virtual memory (RAM + swap space).
The limit for automatic variables is the amount of memory allocated for the machine stack. 10MB is actually rather high; 1 or 2 MB is a more common default.
Obviously, the 3GB is the OS limit -- it's roughly the size of the process space allowed by the OS to a program. It'll vary widely by OS and hardware platform.
The 3 GB limit can probably be fixed by moving to a 64-bit OS (with plenty of RAM).
There's a reasonable chance (but no certainty) that the 10 MB limit can be adjusted with some linker flags.
I just executed a program that mallocs 13 MB in a 12 MB machine (QEMU Emulated!) . Not just that, i even browsed through the memory and filled junk in it...
void
large_mem(void)
{
#define LONGMEM 13631488
long long *ptr = (long long *)malloc(LONGMEM);
long long i;
if(!ptr) {
printf("%s(): array allocation of size %lld failed.\n",__func__,LONGMEM);
ASSERT(0);
}
for(i = 0 ; i < LONGMEM ; i++ ) {
*(ptr+i)=i;
}
free(ptr);
}
How is it possible ? I was expecting a segmentation fault.
It's called virtual memory which is allocated for your program. It's not real memory which you call RAM.
There is a max limit for virtual memory as well, but it's higher than RAM. It's implemented (and defined) by your operating system.
This is called as Lazy Allocation.
Most OS like Linux have an Lazy Allocation memory model wherein the returned memory address is a virtual address and the actual allocation only happens at access-time. The OS assumes that it will be able to provide this allocation at access-Time.
The memory allocated by malloc is not backed by real memory until the program actually touches it.
While, since calloc initializes the memory to 0 you can be assured that the OS has already backed the allocation with actual RAM (or swap).
Try using callocand most probably it will return you out of memory unless your swap file/partition is big enough to satisfy the request.
Sounds like your operating system is swapping pages:
Paging is an important part of virtual memory implementation in most
contemporary general-purpose operating systems, allowing them to use
disk storage for data that does not fit into physical random-access
memory (RAM).
In other words, the operating system is using some of your hard disk space to satisfy your 13 MB allocation request (at great expense of speed, since the hard disk is much, much slower than RAM).
Unless the virtualized OS has swap available, what you're encountering is called overcommit, and it basically exists because the easy way to manage resources in a system with virtual memory and demand/copy-on-write pages is not to manage them. Overcommit is a lot like a bank loaning out more money than it actually has -- it seems to work for a while, then things come crashing down. The first thing you should do when setting up a Linux system is fix this with the command:
echo "2" > /proc/sys/vm/overcommit_memory
That just affects the currently running kernel; you can make it permanent by adding a line to /etc/sysctl.conf:
vm.overcommit_memory=2