TCMalloc memory leak debugging - c++
I've compiled an application with tcmalloc and used HEAPPROFILE environment variable to get heap files
every 10MB or so a new heap file is created and according to the tcmalloc page i can use the pprof tool to compare heap files, and see what are the additional non released objects (possible leak).
pprof --text myapp --base=myapp.0001.heap myapp.0047.heap
the result is:
...
Total: 4600.7 MB
4592.3 99.8% 99.8% 4592.3 99.8% 0x00000000009f1d25
7.3 0.2% 100.0% 7.3 0.2% 0x00000000009f1cfc
1.0 0.0% 100.0% 1.0 0.0% 0x00000000009f74f1
0.0 0.0% 100.0% 4600.7 100.0% 00007f07fe149b44
0.0 0.0% 100.0% 4600.7 100.0% 0x0000000000480da1
0.0 0.0% 100.0% 4600.7 100.0% 0x00000000004b5a3e
0x00000000009f1d25 is a nice address, but i can't really do anything with this data.
I've tried running the same in an helloworld application
pprof --text helloworld helloworld.0001.heap
Using local file helloworld.
Using local file helloworld.0001.heap.
Total: 9.5 MB
9.5 100.0% 100.0% 9.5 100.0% BigNumber::BigNumber
0.0 0.0% 100.0% 0.0 0.0% __GI__IO_file_doallocate
0.0 0.0% 100.0% 9.5 100.0% main
0.0 0.0% 100.0% 0.0 0.0% _IO_new_file_overflow
0.0 0.0% 100.0% 0.0 0.0% _IO_new_file_xsputn
0.0 0.0% 100.0% 0.0 0.0% __GI__IO_doallocbuf
0.0 0.0% 100.0% 0.0 0.0% __GI__IO_fwrite
0.0 0.0% 100.0% 9.5 100.0% __libc_start_main
0.0 0.0% 100.0% 9.5 100.0% _start
0.0 0.0% 100.0% 0.0 0.0% std::__ostream_insert
0.0 0.0% 100.0% 0.0 0.0% std::operator<<
Here we can see clearly that all the functions have clear names, and the leak is from the BigNumber constructor.
Can anyone point me in the right direction toward getting the meaning of the address above ?
Related
How shoud the results from cachegrind be interpretated?
I need to profile a program in development to understand what bottlenecks there may be and in particular whether there are any due to memory accesses. To do this I used cachegrind built into valgrind. I compiled the program using gcc and the -g flag after which I ran valgrind using the command valgrind --tool=cachegrind ./a.out. The result printed on the terminal was as follows: ==11611== Cachegrind, a cache and branch-prediction profiler ==11611== Copyright (C) 2002-2017, and GNU GPL'd, by Nicholas Nethercote et al. ==11611== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info ==11611== Command: ./profiling ==11611== --11611-- warning: L3 cache found, using its data for the LL simulation. Elapsed computation time: 33.46223 seconds ==11611== ==11611== I refs: 10,918,854,735 ==11611== I1 misses: 1,655 ==11611== LLi misses: 1,646 ==11611== I1 miss rate: 0.00% ==11611== LLi miss rate: 0.00% ==11611== ==11611== D refs: 4,620,671,815 (4,235,254,268 rd + 385,417,547 wr) ==11611== D1 misses: 3,222,370 ( 2,887,833 rd + 334,537 wr) ==11611== LLd misses: 18,506 ( 16,679 rd + 1,827 wr) ==11611== D1 miss rate: 0.1% ( 0.1% + 0.1% ) ==11611== LLd miss rate: 0.0% ( 0.0% + 0.0% ) ==11611== ==11611== LL refs: 3,224,025 ( 2,889,488 rd + 334,537 wr) ==11611== LL misses: 20,152 ( 18,325 rd + 1,827 wr) ==11611== LL miss rate: 0.0% ( 0.0% + 0.0% ) The thing I don't understand is the final percentage for LL miss rate, in fact doing LL misses/LL refs * 100 should come 0.6% while the terminal reports 0.0%. Is it an approximation done by cachegrind ? Using kcachegrind I only get percentages next to the event types and next to the lines of code (as in figure). Is it possible to see the number of misses instead ?
MAC OS: WARNING: Project ERROR: Unknown module(s) in QT: webkitwidgets
I am trying to build notepadqq for MacOS following the official doc. while running the make command I am facing below issue - $ make cd src/ui/ && ( test -e Makefile || /usr/local/Cellar/qt/6.3.0/bin/qmake -o Makefile /Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/ui.pro PREFIX= QMAKE_CXX=c++ 'QMAKE_CXXFLAGS= ' QMAKE_LFLAGS= LRELEASE=/usr/local/Cellar/qt/6.3.0/bin/lrelease CONFIG+= ) && /Library/Developer/CommandLineTools/usr/bin/make -f Makefile Project MESSAGE: Release build WARNING: Project ERROR: Unknown module(s) in QT: webkitwidgets Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_de.qm'... Generated 348 translation(s) (348 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_es.qm'... Generated 365 translation(s) (365 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_fr.qm'... Generated 345 translation(s) (345 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_hu.qm'... Generated 453 translation(s) (453 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_it.qm'... Generated 345 translation(s) (345 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_ja.qm'... Generated 344 translation(s) (344 finished and 0 unfinished) Ignored 1 untranslated source text(s) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_pl.qm'... Generated 348 translation(s) (348 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_pt.qm'... Generated 345 translation(s) (345 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_ru.qm'... Generated 328 translation(s) (326 finished and 2 unfinished) Ignored 17 untranslated source text(s) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_sl.qm'... Generated 345 translation(s) (345 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_sv.qm'... Generated 359 translation(s) (359 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_uk.qm'... Generated 365 translation(s) (365 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_zh.qm'... Generated 363 translation(s) (363 finished and 0 unfinished) Project ERROR: Unknown module(s) in QT: webkitwidgets make: *** [sub-src-ui-make_first] Error 3 Any suggestions on how i can fix the issue? Any help is greatly appreciated (working answer will be accepted or up voted). I am using MacOS Monterey - 12.4 (21F79) As can be seen in the log - i have installed QT(qt/6.3.0) using brew by following the link. Like Parisa Adviced, i tried with QT5, but same story. C02XP51WJG5J:notepadqq-1.4.0 vsamula$ ./configure checking for QT5 qmake... /usr/local/Cellar/qt#5/5.15.3/bin/qmake checking for lrelease... /usr/local/Cellar/qt#5/5.15.3/bin/lrelease checking for c++... /usr/bin/c++ checking whether c++ compiler builds test program... ok checking whether c++ compiler supports -std=c++0x... ok checking whether compiled test program works... ok checking for make... /usr/bin/make checking for pkg-config... /usr/local/bin/pkg-config checking for Qt5Core library... not found! checking for Qt5Gui library... not found! checking for Qt5Network library... not found! checking for Qt5WebKit library... not found! checking for Qt5Widgets library... not found! checking for Qt5WebKitWidgets library... not found! checking for Qt5PrintSupport library... not found! checking for Qt5Svg library... not found! generate Makefile... Info: creating stash file /Users/vsamula/Downloads/notepadqq-1.4.0/.qmake.stash done C02XP51WJG5J:notepadqq-1.4.0 vsamula$ make cd src/ui/ && ( test -e Makefile || /usr/local/Cellar/qt#5/5.15.3/bin/qmake -o Makefile /Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/ui.pro PREFIX= QMAKE_CXX=c++ 'QMAKE_CXXFLAGS= ' QMAKE_LFLAGS= LRELEASE=/usr/local/Cellar/qt#5/5.15.3/bin/lrelease CONFIG+= ) && /Library/Developer/CommandLineTools/usr/bin/make -f Makefile Project MESSAGE: Release build WARNING: Project ERROR: Unknown module(s) in QT: webkitwidgets Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_de.qm'... Generated 348 translation(s) (348 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_es.qm'... Generated 365 translation(s) (365 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_fr.qm'... Generated 345 translation(s) (345 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_hu.qm'... Generated 453 translation(s) (453 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_it.qm'... Generated 345 translation(s) (345 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_ja.qm'... Generated 344 translation(s) (344 finished and 0 unfinished) Ignored 1 untranslated source text(s) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_pl.qm'... Generated 348 translation(s) (348 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_pt.qm'... Generated 345 translation(s) (345 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_ru.qm'... Generated 328 translation(s) (326 finished and 2 unfinished) Ignored 17 untranslated source text(s) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_sl.qm'... Generated 345 translation(s) (345 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_sv.qm'... Generated 359 translation(s) (359 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_uk.qm'... Generated 365 translation(s) (365 finished and 0 unfinished) Updating '/Users/vsamula/Downloads/notepadqq-1.4.0/src/ui/../translations/notepadqq_zh.qm'... Generated 363 translation(s) (363 finished and 0 unfinished) Project WARNING: Qt has only been tested with version 11.0 of the platform SDK, you're using . Project WARNING: This is an unsupported configuration. You may experience build issues, and by using Project WARNING: the 12.3 SDK you are opting in to new features that Qt has not been prepared for. Project WARNING: Please downgrade the SDK you use to build your app to version 11.0, or configure Project WARNING: with CONFIG+=sdk_no_version_check when running qmake to silence this warning. Project ERROR: Unknown module(s) in QT: webkitwidgets make: *** [sub-src-ui-make_first] Error 3
It is definitely a version issue. You should try build from a 5.12 source say from https://download.qt.io/archive/qt/5.12/ with version 5.12.8 and lower. The steps for building from source for a mac is at https://doc.qt.io/qt-5/macos-building.html.
How do you read gcno and gdna files to see test coverage?
I am using gcovr to try to see how much my tests cover the codebase. In fact I have a single super small test so I am expecting gcovr to return a very small number, probably less than a percent. I compiled with the --coverage flag enabled in every single compilation unit and the -lgcov flag active on ever single linking command. Then i ran both the executable and the tests and generated the gcno and gdna files. Then I do: gcovr --root ../Src/ ../Src/ ./GeometryTests.p ------------------------------------------------------------------------------ GCC Code Coverage Report Directory: ../Src/ ------------------------------------------------------------------------------ File Lines Exec Cover Missing ------------------------------------------------------------------------------ Engine/Geometry/GeometryUtils.hpp 18 18 100% Engine/Geometry/tests/GeometryTests.cpp 3 3 100% Engine/Geometry/tests/TestsGeometryUtils.cpp 18 18 100% ------------------------------------------------------------------------------ TOTAL 39 39 100% ----------------------------------------------------------------------------- Well this isn't right, I have dozens of files and my tiny test only tests 2 functions, this is clearly wrong. I then tried: gcovr --root ../Src/ ../Src/ ./GeometryTests.p ./SponzaScene.p ------------------------------------------------------------------------------ GCC Code Coverage Report Directory: ../Src/ ------------------------------------------------------------------------------ File Lines Exec Cover Missing ------------------------------------------------------------------------------ Engine/Animation/GltfLib.hpp 1 1 100% Engine/Animation/Image/CpuImage.hpp 1 1 100% Engine/Geometry/GeometryUtils.hpp 18 18 100% Engine/Geometry/HMesh.hpp 592 0 0% 37,87-89,92-94,96-98,100-101,103,105-107,110,114-118,121,123-125,127,129,132-133,135,137-139,141,143,146,148-150,174-176,180,182-184,186,188-190,192,194-196,198,200-201,203,205-206,208,210,212-213,215,217-218,220,222-223,225,227-228,230,232-233,236,238-239,241,243-244,246,248-249,251,253-254,256,258-259,261,263,265,267-268,271,273,275,277,280,282,285,291,297-298,309,311,313,315,317-318,320,322-324,326-327,329,332,334,336-343,345,348,350-351,354,357,359-360,362,365,367-368,370,373,375-376,378,381,383-384,386,398,410,413-418,420,429-432,434-437,439-441,445,559,561,563-564,566-571,573-574,577-578,580,584,587-588,590-591,593,595-596,598-600,604-605,609,612,614,616-618,620,622-625,631,635-641,643-648,651-653,657-659,661-662,665-670,672-675,677-678,682-684,686-690,692-695,697-699,701-704,706-707,709-711,715,718-719,721-725,727-729,732,737,741-745,747-750,753-755,757-758,760,763-768,770-772,774,778-780,782-788,790-792,794-796,798-800,802,804,808,811-813,815-821,823-828,830-832,834-836,838-840,842-844,846-848,850-852,854-856,858-860,865,867-868,870-871,873,876,882,885-886,888,890-892,895,897,900,913,916-917,919-920,924,926-927,931,933-934,938,941-943,945,947-949,953,955-956,961,969-973,975-976,978,980-983,986,988,990-992,994,996-997,999,1002-1003,1007,1009-1010,1015-1026,1028,1030,1032,1034-1036,1039,1041-1042,1044-1045,1048,1050,1052-1057,1059,1062,1067-1068,1070-1071,1075,1079-1080,1082,1084-1086,1088,1090-1091,1094-1096,1099-1101,1104-1108,1113,1115-1117,1119-1121,1125,1127,1130-1131,1135,1138,1141,1145-1146,1148-1151,1153-1155,1157,1162,1164-1170,1175,1177-1183,1188,1191-1192,1194,1196,1198-1200,1202-1204,1206-1208,1210-1213,1215-1217,1220,1222,1228,1230-1231,1233-1236,1238-1239,1241-1243,1246-1248,1252,1254-1259,1263,1266-1268,1270-1273,1275,1279,1281-1282,1285-1286 Engine/Geometry/tests/GeometryTests.cpp 3 3 100% Engine/Geometry/tests/TestsGeometryUtils.cpp 18 18 100% Engine/Helpers/EigenHelpers.hpp 4 0 0% 187,189,192,195 Engine/Renderer/Camera/Camera.hpp 13 12 92% 46 Engine/Renderer/IO/Window.hpp 3 3 100% Engine/Renderer/Renderer.hpp 7 7 100% Engine/Renderer/Rendering/Effects.hpp 35 35 100% Engine/Renderer/Rendering/Gallery.hpp 82 78 95% 294-296,298 Engine/Renderer/Rendering/RenderTargetStorage.hpp 1 1 100% Engine/Renderer/Rendering/VulkanLayer/Image.hpp 1 1 100% Engine/Renderer/Rendering/VulkanLayer/Memory.hpp 3 3 100% Engine/Renderer/Rendering/VulkanLayer/RenderTarget.hpp 1 1 100% Engine/Renderer/Rendering/VulkanLayer/RenderingPipeline.hpp 3 3 100% Engine/Renderer/Rendering/VulkanLayer/ShaderProgram.hpp 1 1 100% Engine/Renderer/Rendering/VulkanLayer/Utils.hpp 5 5 100% Engine/Shared/Shared.hpp 8 3 37% 71,73,181,184-185 examples/SponzaScene/sponza_scene.cpp 180 172 95% 236,240,242-243,245-246,341,343 ------------------------------------------------------------------------------ TOTAL 980 366 37% ------------------------------------------------------------------------------ At least it is not 100% but I can guarantee that my test doesn't touch most of the files and i don't quite understand how I should be setting up the compilation, linking and gcovr invocation to analyze test coverage.
Looks like previous results were not cleaned. Try to compile with -ftest-coverage Maybe lcov + htmlgen will works for you. Here is an example: rm -f *.gcno *.gcda coverage.info coverage_dir gcc -I${INCLUDE_PATH} -fprofile-arcs -ftest-coverage -O0 --coverage main.c -o test-coverage rm -f *.gcda coverage.info ./test-coverage lcov --capture --rc lcov_branch_coverage=1 --directory . --config-file ./lcovrc --output coverage.info genhtml --branch-coverage --output ./coverage_dir coverage.info
GCC build fails on OSX 10.14.1 with "malformed object (unknown load command 1)"
I am trying to build a library on my mac (OSX Version 10.14.1, Xcode Version 10.1) and it fails giving following error: Building target: application.a ar -r "application.a" api.o ar: creating archive application.a /opt/local/bin/ranlib: object: application.a(api.o) malformed object (unknown load command 1) ar: internal ranlib command failed make[1]: *** [application.a] Error 1 make: *** [all] Error 2 by running 'otool -l lib/api.o' I get the following result: Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags 0xfeedfacf 16777223 3 0x00 1 5 616 0x00002000 Load command 0 cmd LC_SEGMENT_64 cmdsize 472 segname vmaddr 0x0000000000000000 vmsize 0x0000000000004fb0 fileoff 648 filesize 20400 maxprot 0x00000007 initprot 0x00000007 nsects 5 flags 0x0 Section sectname __text segname __TEXT addr 0x0000000000000000 size 0x00000000000040ae offset 648 align 2^4 (16) reloff 21048 nreloc 329 flags 0x80000400 reserved1 0 reserved2 0 Section sectname __cstring segname __TEXT addr 0x00000000000040ae size 0x00000000000000cb offset 17206 align 2^0 (1) reloff 0 nreloc 0 flags 0x00000002 reserved1 0 reserved2 0 Section sectname __const segname __TEXT addr 0x0000000000004180 size 0x000000000000003f offset 17416 align 2^4 (16) reloff 0 nreloc 0 flags 0x00000000 reserved1 0 reserved2 0 Section sectname __compact_unwind segname __LD addr 0x00000000000041c0 size 0x00000000000005e0 offset 17480 align 2^3 (8) reloff 23680 nreloc 47 flags 0x02000000 reserved1 0 reserved2 0 Section sectname __eh_frame segname __TEXT addr 0x00000000000047a0 size 0x0000000000000810 offset 18984 align 2^3 (8) reloff 0 nreloc 0 flags 0x6800000b reserved1 0 reserved2 0 Load command 1 cmd ?(0x00000032) cmdsize 24 Load command 2 cmd LC_DATA_IN_CODE cmdsize 16 dataoff 24056 datasize 8 Load command 3 cmd LC_SYMTAB cmdsize 24 symoff 24064 nsyms 93 stroff 25552 strsize 1668 Load command 4 cmd LC_DYSYMTAB cmdsize 80 ilocalsym 0 nlocalsym 15 iextdefsym 15 nextdefsym 39 iundefsym 54 nundefsym 39 tocoff 0 ntoc 0 modtaboff 0 nmodtab 0 extrefsymoff 0 nextrefsyms 0 indirectsymoff 0 nindirectsyms 0 extreloff 0 nextrel 0 locreloff 0 nlocrel 0 I am guessing load command miss matches between my gcc and osx version, but I can't find 0x00000032 command on mach-o/loader.h. does anyone have a clue about this? here is the 'otool --version': llvm-otool(1): Apple Inc. version cctools-895 LLVM (http://llvm.org/): LLVM version 4.0.1 Optimized build. Default target: x86_64-apple-darwin18.2.0 Host CPU: haswell Registered Targets: aarch64 - AArch64 (little endian) aarch64_be - AArch64 (big endian) amdgcn - AMD GCN GPUs arm - ARM arm64 - ARM64 (little endian) armeb - ARM (big endian) bpf - BPF (host endian) bpfeb - BPF (big endian) bpfel - BPF (little endian) hexagon - Hexagon lanai - Lanai mips - Mips mips64 - Mips64 [experimental] mips64el - Mips64el [experimental] mipsel - Mipsel msp430 - MSP430 [experimental] nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit ppc32 - PowerPC 32 ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE r600 - AMD GPUs HD2XXX-HD6XXX riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V sparc - Sparc sparcel - Sparc LE sparcv9 - Sparc V9 systemz - SystemZ thumb - Thumb thumbeb - Thumb (big endian) x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 xcore - XCore
Do you have macports installed? I had the same issue and it seems that macports cctool package install /opt/local/bin/otool which is likely ahead in your path. I suspect that /usr/bin/otool gives the right one ...
Why the size of wxWidget libs ver. 2.9.1 is much bigger than ver. 2.8.12?
I build last version of wxWidgets and got a huge libraries sizes, for example in compare with 2.8.12 version: MinGW (UNICODE=1 SHARED=0 MONOLITHIC=0 BUILD=release) ver. 2.9.1 libwxmsw29u_core.a 16 840 686 bytes minimal 7 448 383 bytes ver. 2.8.12 libwxmsw28u_core.a 8 444 360 bytes minimal 3 108 575 bytes MSVC-10 ver. 2.9.1 wxmsw29u_core.lib 64 806 104 bytes minimal 3 992 576 bytes ver. 2.8.12 wxmsw28_core.lib 31 135 714 bytes minimal 1 187 840 bytes The size grown up over than 2.3 times for MinGW and over than 3.6 times for MSVC-10. I heared, that 2.9 branch has assertions and some debug info even in release version, so I set DEBUG_FLAG to 0 in config.gcc, in include/msw/setup.h uncommented #ifdef NDEBUG #define wxDEBUG_LEVEL 0 #else #define wxDEBUG_LEVEL 2 #endif and used mingw32-make UNICODE=1 SHARED=0 DEBUG_FLAG=0 BUILD=release MONOLITHIC=0 command to build the library. In make log compiller has -O2 optimization keys and NDEBUG defined, but the size didn't reduce much: libwxmsw29u_core.a 14 555 082 bytes minimal 5 856 718 What happend with wxWidget in 2.9 branch? Why the size is so huge? What I'm doing wrong?