Allocate large array in PGI Fortran - fortran

I am trying to allocate an real array finn_var(459,299,27,24,nspec) in Fortran. nspec = 24 works ok, while nspec = 25 not. No error message for allocation process, but print command print empty rather than zero values. If you use the array, there will be "segmentation fault" error message. The test program is
program test
implicit none
integer :: nx, ny, nez, nt, nspec
integer :: allocation_status
real , allocatable :: finn_var(:,:,:,:,:)
nx = 459
ny = 299
nez = 27
nt = 24
nspec = 24
allocate( finn_var(nx, ny, nez, nt, nspec), stat = allocation_status )
if (allocation_status > 0) then
print*, "Allocation error for finn_var"
stop
end if
print*, finn_var
end
Should not be the memory issue. I allocated double precision finn_var(459,299,27,24,24) without problem. What is the reason then?
I use pgf90 on linux server. the cat /proc/meminfo command:
MemTotal: 396191724 kB
MemFree: 66065188 kB
Buffers: 402388 kB
Cached: 274584600 kB
SwapCached: 0 kB
Active: 131679328 kB
Inactive: 191625200 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 396191724 kB
LowFree: 66065188 kB
SwapTotal: 20971484 kB
SwapFree: 20971180 kB
Dirty: 605508 kB
Writeback: 0 kB
AnonPages: 48317148 kB
Mapped: 123328 kB
Slab: 6612824 kB
PageTables: 132920 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 219067344 kB
Committed_AS: 53206972 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 275624 kB
VmallocChunk: 34359462559 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
the unlimit -a command:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3153920
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 3153920
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I compiled by pgf90. But if I compiled by gfortran, there is no problem.

It doesn't have to be insufficient memory. The size of the array is 2 223 304 200. That is suspiciously close to the maximum 32bit integer 2 147 483 648.
It looks like that the element count that the compiler uses internally overflows. The internal call to malloc requests not enough memory and then any attempt to read some of the elements at the end fails.
It is a limitation of the compiler in its default settings. It can be set-up to use 64bit addressing by using the option ‑Mlarge_arrays.
See http://www.pgroup.com/products/freepgi/freepgi_ref/ch05.html#ArryIndex

Your problem is most likely a memory issue.
You array demands 459*299*27*24 * 4B per nspec (assuming default real requires 4B of memory). For nspec == 24 this results in a memory requirement of approximately 7.95GiB, while nspec == 25 needs around 8.28GiB.
I guess, your physical memory is limited to 8GiB or some ulimit limits the amount of allowed memory for this process.

Related

Determine if an allocation via malloc() is backed by a huge page

I understand pretty well how transparent hugepages work, and that any allocation, such as those performed by malloc may be satisfied by a huge page.
What I'd like to know, is if there is any check I can make (possibly heuristic) after an allocation to determine if the memory is backed by a huge page.
You can determine the exact status of any page, including whether it is backed by a transparent (or non-transparent) hugepage by looking up the "pfn" (page frame number) in the /proc/kpageflags file. You get the pfn for a page by reading from the /proc/$PID/pagemap file for your process, which is indexed by virtual address.
Unfortunately, both the pfn value from pagemap1 and the entire /proc/kpageflags file are accessible only to root users. Still if you can run your process as root at least in the testing or benchmarking scenario you are interested in, this works well.
I wrote a small library called page-info which does the relevant parsing for you. Give it a range of memory and it will return you info on each page, including whether it is present in memory, backed by a hugepage, etc.
For example, running the included test process as sudo ./page-info-test THP gives the following output:
PAGE_SIZE = 4096, PID = 18868
size memset FLAG SET UNSET UNAVAIL
0.25 MiB BEFORE THP 0 1 64
0.25 MiB AFTER THP 0 65 0
0.50 MiB BEFORE THP 0 1 128
0.50 MiB AFTER THP 0 129 0
1.00 MiB BEFORE THP 0 1 256
1.00 MiB AFTER THP 0 257 0
2.00 MiB BEFORE THP 0 1 512
2.00 MiB AFTER THP 0 513 0
4.00 MiB BEFORE THP 0 1 1024
4.00 MiB AFTER THP 512 513 0
8.00 MiB BEFORE THP 0 1 2048
8.00 MiB AFTER THP 1536 513 0
16.00 MiB BEFORE THP 0 1 4096
16.00 MiB AFTER THP 3584 513 0
32.00 MiB BEFORE THP 0 1 8192
32.00 MiB AFTER THP 7680 513 0
64.00 MiB BEFORE THP 0 1 16384
64.00 MiB AFTER THP 15872 513 0
128.00 MiB BEFORE THP 0 1 32768
128.00 MiB AFTER THP 32256 513 0
256.00 MiB BEFORE THP 0 1 65536
256.00 MiB AFTER THP 65024 513 0
512.00 MiB BEFORE THP 0 1 131072
512.00 MiB AFTER THP 124416 6657 0
1024.00 MiB BEFORE THP 0 1 262144
1024.00 MiB AFTER THP 0 262145 0
DONE
The UNAVAIL column means that no information about the mapping was available - usually because the page has never been accesses and so isn't yet backed by any page at all. You can see that for these "largeish" allocations only a single page is mapped in following the allocation, since we haven't touched the memory.
The AFTER rows are the same information after calling memset() on the entire allocation, which causes all pages to be physically allocated. Here we can see that no allocations are backed by transparent hugepages until we hit allocations of 4 MiB, at which point the majority of each allocation is backed by THP, except for 513 pages (which turn out to be at the edges of the allocated region). At 512 MiB the system starts running out of available hugepages but still satisfies most of the allocation, but at 1024 MiB the entire allocation is satisfied with small pages.
This library isn't production ready so don't use it for anything critical (e.g., some failures simply call exit()). Contributions welcome.
1 Since kernel 4.0 approximately, before that the pfn was accessible to non-root user processes. From 4.0 to 4.1 or thereabouts, the entire pagemap was off-limits to non-root processes, but since then the file is again available but with the pfn masked out (it will always appear as zero).
There is a difference between traditional hugepages and transparent huge pages (THP). In the case of THP's, the application can use huge pages without any developer support (mmap, shmget, etc) or sys-admin intervention.
In the code, I am afraid there may be no straight forward way check this. However, if you know the sizeof() allocated data structure or buffers, it worth grepping and checking the THP usage on the system using the following command. This usage should increase while running your application:
# grep AnonHugePages /proc/meminfo
AnonHugePages: 2648064 kB

Cache line size

It might be a very common and simple question but I need some explanation about the curve that I just obtained from a cache benchmarks code. The goal here is to find the cache line size. I used the code from here:
(h**ps://github.com/jiewmeng/cs3210-assign1/blob/master/cache-l1-line.cpp)
This is the curve that I have obtained from running the code on my machine (Macbook Pro with core i7 - cache line size is 64byte - L1 data cache is 32KB).
The Time vs different stride size curve
I think the peak happens on 128 bytes and not on the 64 bytes. if it is true I want to know why?
Why the time is reduced at 512 bytes?
Update:
I also ran a code to determine the size of the L1 and L2 caches. Here is the figure just to document the data. As you can see there is two peak in 32KB (L1 Cache size) and 256KB (L2 Cache size).
Question:
I am wondering if there is any way to find the size of L3 shared cache.
Cache size figure.
Thanks
I'm guessing that the 128B peak is most likely due to spatial prefetching. You can see in Intels' Optimization guide, under section 2.1.5.4
This prefetcher strives to complete every cache line fetched to the L2 cache with the pair line that completes it to a 128-byte aligned chunk
It wouldn't be a clean jump since this prefetches is not always firing, and even when it does, it only prefetches into the L2, but it's much better than fetching from memory. To make sure this is the case, you can disable prefetches (through BIOS or other means, although some systems may not support that), and check again.
As for the L3 size - you didn't specify your exact model, but i'm guessing you have more than 4M L3 - just keep the curve going and see if it jumps.
EDIT
Just noticed another thing - your k*i expression is probably overflowing int at the max range, which means your access pattern might not be cyclic as you expect.
My BusSpeed benchmark was intended to identify cache sizes and performance at different strides, to show burst reading on buses:
http://www.roylongbottom.org.uk/busspd2k%20results.htm
Following are results on a Core i7 with 8 MB L3:
Memory Reg2 Reg2 Reg2 Reg2 Reg1 Reg2 Reg1 Reg2 Reg1 Reg8
KBytes Inc64 Inc32 Inc16 Inc8 Inc4 Inc4 Inc4 Inc4 Inc8 Inc8
Used MB/S MB/S MB/S MB/S MB/S MB/S MB/S MB/S MB/S MB/S
4 10025 10800 11262 11498 11612 11634 5850 11635 23093 23090
8 10807 11267 11505 11627 11694 11694 5871 11694 23299 23297
16 11251 11488 11620 11614 11712 11719 5873 11718 23391 23398
32 9893 9853 10890 11170 11558 11492 5872 11466 21032 21025
64 3219 4620 7289 9479 10805 10805 5875 10797 14426 14426
128 3213 4805 7305 9467 10811 10810 5875 10805 14442 14408
256 3144 4592 7231 9445 10759 10733 5870 10743 14336 14337
512 2005 3497 5980 9056 10466 10467 5871 10441 13906 13905
1024 2003 3482 5974 9017 10468 10466 5874 10467 13896 13818
2048 2004 3497 5958 9088 10447 10448 5870 10447 13857 13857
4096 1963 3398 5778 8870 10328 10328 5851 10328 13591 13630
8192 1729 3045 5322 8270 9977 9963 5728 9965 12923 12892
16384 692 1402 2495 4593 7811 7782 5406 7848 8335 8337
32768 695 1406 2492 4584 7820 7826 5401 7792 8317 8322
65536 695 1414 2488 4584 7823 7826 5403 7800 8321 8321
131072 696 1402 2491 4575 7827 7824 5411 7846 8322 8323
262144 696 1413 2498 4594 7791 7826 5409 7829 8333 8334
524288 693 1416 2498 4595 7841 7842 5411 7847 8319 8285
1048576 704 1415 2478 4591 7845 7840 5410 7853 8290 8283
End of test Fri Jul 30 16:44:29 2010
CPUID and RDTSC Assembly Code
CPU GenuineIntel, Features Code BFEBFBFF, Model Code 000106A5
Intel(R) Core(TM) i7 CPU 930 # 2.80GHz Measured 2807 MHz

Calculating used memory by a set of processes on Linux

I'm having trouble with calculating the actually used memory (resident) by a set of processes.
The issue that just came up is a user with a set of processes that share memory between themselves, so a simple addition of used memory ends up with a nonsense number (>60gb when the machine only has 48gb memory).
Is there any simple way to approach this problem?
I can probably do some approximation. Take (res mem - shared mem) * num proc + shared mem. But not all processes necessarily share the same memory block.
I'm looking for a POSIX or Linux solution to this problem for C/C++.
You will want to iterate through each processes /proc/[pid]/smaps
It will contain an entry for each VM mapping of the likes:
7ffffffe7000-7ffffffff000 rw-p 00000000 00:00 0 [stack]
Size: 100 kB
Rss: 20 kB
Pss: 20 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 20 kB
Referenced: 20 kB
Anonymous: 20 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Private_Dirty memory is what you are interested in.
If you have the Pss field in your smaps file then this is the amount of resident memory divided by the amount of processes that share the physical memory.
Private_Clean could be copy-on-write mappings. Those are commonly used for shared libraries and are generally read/no-write/execute.

Diagnosing ENOMEM with popen() and system() calls in C++

I'm dealing with a gargantuan C++ code for computational physics (that I didn't write) which calls other executables using system() calls. Sometimes in the middle of a simulation these system() calls will fail even for simple calls like system("echo something); When they fail they return immediately with return value -1.
I created a version of the code that uses popen() instead of system() to launch these other executables. In this version, popen() fails and errno is set to 12 (ENOMEM).
This is running on a machine with 96GB of RAM running CentOS 6.3 (via ROCKS 6.1) through the Torque PBS system.
Note that this behavior is somewhat uncommon, but it appears to happen for simulations which use large amounts of memory -- but far less than the amount of memory that is available.
I currently have a simulation running which is exhibiting this behavior. It is attempting to make a system() call every 30 seconds and failing, which allows me to monitor the OS memory resources. The contents of /proc/meminfo are
MemTotal: 99195180 kB
MemFree: 1758804 kB
Buffers: 14612 kB
Cached: 46502432 kB
SwapCached: 7004 kB
Active: 60758772 kB
Inactive: 35238760 kB
Active(anon): 45458924 kB
Inactive(anon): 4024068 kB
Active(file): 15299848 kB
Inactive(file): 31214692 kB
Unevictable: 9752 kB
Mlocked: 9752 kB
SwapTotal: 1023992 kB
SwapFree: 999432 kB
Dirty: 16 kB
Writeback: 8 kB
AnonPages: 49483620 kB
Mapped: 10292 kB
Shmem: 8 kB
Slab: 235356 kB
SReclaimable: 193468 kB
SUnreclaim: 41888 kB
KernelStack: 2120 kB
PageTables: 99536 kB
NFS_Unstable: 4 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 50621580 kB
Committed_AS: 49576180 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 482936 kB
VmallocChunk: 34307833876 kB
HardwareCorrupted: 8 kB
AnonHugePages: 43315200 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 5568 kB
DirectMap2M: 2082816 kB
DirectMap1G: 98566144 kB
The contents of /proc/5939/status (which is the process in question) is
Name: BAD_EXECUTABLE
State: S (sleeping)
Tgid: 5939
Pid: 5939
PPid: 5938
TracerPid: 0
Uid: 505 505 505 505
Gid: 505 505 505 505
Utrace: 0
FDSize: 256
Groups: 426 505 801
VmPeak: 49733876 kB
VmSize: 49482532 kB
VmLck: 0 kB
VmHWM: 49721496 kB
VmRSS: 49470248 kB
VmData: 49481080 kB
VmStk: 128 kB
VmExe: 1316 kB
VmLib: 0 kB
VmPTE: 96656 kB
VmSwap: 10624 kB
Threads: 1
SigQ: 0/774828
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000000000000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed: ffffff
Cpus_allowed_list: 0-23
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 57003
nonvoluntary_ctxt_switches: 1057385
I'm at a bit of a loss at how to debug this issue, especially since I can't recreate it with a smaller simulation. My simulation says it's using 47GB of memory, while /proc/meminfo shows that less than 2GB of 96GB of memory is free, and there shouldn't be anything else running that uses tens of GBs of memory.
This forum seems to indicate that previous memory errors could have corrupted the heap. Is this a valid possibility? What else could I look at that would help me narrow down this issue?

Need to get process memory using c++

I want to calculate my process memory (rss) at runtime in my application (c++/unix/multithreaded).Do we have any API to use for that.Please note that , I am aware of reading /proc/stat and getrusage() , but dont want to read/parse a system file from appication and getrusage() does not work in my linux distribution.
The whole intent was to check for memory leak caused by my application . I have even tried tracking memory by overloading new/malloc/calloc/realloc and get the memory allocation trakced, but even with thsese I am not able to track the whole memory allocated by process. It would be also helpfull if you can suggest the other probable areas where I should look for memory allocation/ memory leak other than the above stated APIs.
I am aware of Valgrind/mpatrol type of memory monitor tools .. but unfortunately it does not work with my application..
Thanks in advance
First, this kind of information is operating system specific. It has to be done differently on Linux, on MacOSX, on FreeBSD...
On Linux, the blessed way, is as every one told you, to use the /proc file system, which is how all the system utilities (e.g. top or ps) are retrieving that information (perhaps by using libproc which is just a wrapper around reads of /proc/ files).
Could you explain why reading e.g. /proc/self/statm or /proc/self/stat or /proc/self/status or /proc/self/maps is not possible for you?
Remember that these /proc/files are pseudo-files, and no actual slow I/O operation to disk is involved in reading them. And you have to read them sequentially, seeking (or stat-ing) them does not work.
It seems to me that
long process_size_in_pages(void)
{
long s = -1;
FILE *f = fopen("/proc/self/statm", "r");
if (!f) return -1;
// if for any reason the fscanf fails, s is still -1,
// with errno appropriately set.
fscanf(f, "%ld", &s);
fclose (f);
return s;
}
is the fastest way to retrieve that information. Why can't you do that?
You could use valgrind. By setting it in monitor mode and calling remote method (gdb) monitor full, it would give you the total, allocated, memory at run time. See this page for more information.
You can read /proc/${pid}/status, it looks like
Name: nginx
State: S (sleeping)
SleepAVG: 98%
Tgid: 11884
Pid: 11884
PPid: 11883
TracerPid: 0
Uid: 99 99 99 99
Gid: 99 99 99 99
FDSize: 64
Groups: 99
VmPeak: 23932 kB
VmSize: 23932 kB
VmLck: 0 kB
VmHWM: 4276 kB
VmRSS: 4276 kB
VmData: 3744 kB
VmStk: 88 kB
VmExe: 452 kB
VmLib: 3024 kB
VmPTE: 88 kB
StaBrk: 1a931000 kB
Brk: 1a974000 kB
StaStk: 7fffc224d560 kB
Threads: 1
SigQ: 0/73712
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000040001000
SigCgt: 0000000198016a07
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
Cpus_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000000f
Mems_allowed: 00000000,00000001
You can parse the VmRSS value.