Valgrind tool which is a well known memory analyzing tool reports an Invalid Read in OCIStmtPrepare in Oracle C API Function. This can be observed in several such Oracle C API functions.
Please refer the following stack trace.
According to my observations and understanding the the application creates a buffer of 317 bytes. However when it is passed to Oracle library it does some memory copy using the __intel_new_memcpy function. However the __intel_new_memcpy function copies 320 bytes (which is 8 from 312). The actual allocated memory was 317 bytes.
Could you please confirm whether this behaviour correct? What goes wrong in this?
==22195== Invalid read of size 8
==22195== at 0x68CD2D9: __intel_new_memcpy (in /x02/app/oracle/product/11.2.0/client_1/lib/libclntsh.so.11.1)
==22195== by 0x5D84158: kpurclientparse (in /x02/app/oracle/product/11.2.0/client_1/lib/libclntsh.so.11.1)
==22195== by 0x5D878DE: kpureq (in /x02/app/oracle/product/11.2.0/client_1/lib/libclntsh.so.11.1)
==22195== by 0x5D607FA: OCIStmtPrepare (in /x02/app/oracle/product/11.2.0/client_1/lib/libclntsh.so.11.1)
==22195== by 0x4099E0: DBCursor::Parse(char const*) (OCICPP.C:1020)
==22195== by 0x40CE29: DBCon::NewCursor(char const*, int) (OCICPP.C:753)
==22195== by 0x4047A6: main (main.cpp:59)
==22195== Address 0xa2e7e68 is 312 bytes inside a block of size 317 alloc'd
==22195== at 0x4C26E1C: operator new[](unsigned long) (vg_replace_malloc.c:305)
==22195== by 0x4EBD00F: String::Set(char const*, unsigned int) (String.cpp:544)
==22195== by 0x4EBD169: String::Set(char const*) (String.cpp:512)
==22195== by 0x4EBD188: String::operator=(char const*) (String.cpp:590)
==22195== by 0x404784: main (main.cpp:55)
Related
I'm using ASAN address sanitizer to detect memory issues. When the program stops ASAN complains about the following:
==102121==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 537 byte(s) in 1 object(s) allocated from:
#0 0x75cb48 in operator new(unsigned long) (/home/app+0x75cb48)
#1 0x7dca83 in __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) /opt/rh/devtoolset-7/root/usr/include/c++/7/ext/new_allocator.h:111
#2 0x7ce766 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/basic_string.tcc:1057
#3 0x7cc54d in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long) (/home/app+0x7cc54d)
#4 0x7c1f2a in std::string::reserve(unsigned long) /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/basic_string.tcc:960
#5 0x7fa0a639c6f5 in std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow(int) (/lib64/libstdc++.so.6+0x9b6f5)
Direct leak of 24 byte(s) in 1 object(s) allocated from:
#0 0x75cec8 in operator new(unsigned long, std::nothrow_t const&) (/home/app+0x75cec8)
#1 0x7fa0a635df1d in __cxa_thread_atexit (/lib64/libstdc++.so.6+0x5cf1d)
Indirect leak of 24 byte(s) in 1 object(s) allocated from:
#0 0x75cec8 in operator new(unsigned long, std::nothrow_t const&) (/home/app+0x75cec8)
#1 0x7fa0a635df1d in __cxa_thread_atexit (/lib64/libstdc++.so.6+0x5cf1d)
I've seen on the ASAN page that it can come from the fact the the standard library is statically linked. Although, in my case it is dynamic one.
The application is compiled with devtoolset-7 on RHEL.
Do you have any idea where the leak comes from?
You can get more info than
#0 0x75cb48 in operator new(unsigned long) (/home/app+0x75cb48)
by using llvm-symbolizer.
Download it, and set the environment variable
ASAN_SYMBOLIZER_PATH=/usr/where/ever/the/binary/is
If you are sure that the leak is a false alarm, you can use a suppression file:
create a suppression text file and add to it: leak: __cxa_thread_atexit
Set environment variable
LSAN_OPTIONS=suppressions=path/to/suppr.txt
and then run your app.
http://clang.llvm.org/docs/AddressSanitizer.html#symbolizing-the-reports
Ubuntu 19 running inside Docker
GCC 8.3
Boost 1.69
Valgrind 3.14.0
When the application is shutting down Valgrind reports these 3 issues:
==70== Mismatched free() / delete / delete []
==70== at 0x483997B: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==70== by 0x4870C89: check_free (dlerror.c:202)
==70== by 0x4870C89: check_free (dlerror.c:186)
==70== by 0x4870C89: free_key_mem (dlerror.c:221)
==70== by 0x4870C89: __dlerror_main_freeres (dlerror.c:239)
==70== by 0x4B59711: __libc_freeres (in /usr/lib/x86_64-linux-gnu/libc-2.29.so)
==70== by 0x482E19E: _vgnU_freeres (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_core-amd64-linux.so)
==70== by 0x4A0A3A9: __run_exit_handlers (exit.c:132)
==70== by 0x4A0A3D9: exit (exit.c:139)
==70== by 0x49E9B71: (below main) (libc-start.c:342)
==70== Address 0x4f6a570 is 0 bytes inside a block of size 312 alloc'd
==70== at 0x4838DBF: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==70== by 0x303D6D: boost::detail::make_external_thread_data() (in /build-context/bin/debug/setmatch-tests)
==70== by 0x305424: boost::detail::add_new_tss_node(void const*, boost::shared_ptr<boost::detail::tss_cleanup_function>, void*) (in /build-context/bin/debug/setmatch-tests)
==70== by 0x3054ED: boost::detail::set_tss_data(void const*,
[...]
==70== Invalid free() / delete / delete[] / realloc()
==70== at 0x483997B: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==70== by 0x4870BB4: free_key_mem (dlerror.c:223)
==70== by 0x4870BB4: __dlerror_main_freeres (dlerror.c:239)
==70== by 0x4B59711: __libc_freeres (in /usr/lib/x86_64-linux-gnu/libc-2.29.so)
==70== by 0x482E19E: _vgnU_freeres (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_core-amd64-linux.so)
==70== by 0x4A0A3A9: __run_exit_handlers (exit.c:132)
==70== by 0x4A0A3D9: exit (exit.c:139)
==70== by 0x49E9B71: (below main) (libc-start.c:342)
==70== Address 0x4f6a570 is 0 bytes inside a block of size 312 free'd
==70== at 0x483997B: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==70== by 0x4870C89: check_free (dlerror.c:202)
==70== by 0x4870C89: check_free (dlerror.c:186)
==70== by 0x4870C89: free_key_mem (dlerror.c:221)
==70== by 0x4870C89: __dlerror_main_freeres (dlerror.c:239)
==70== by 0x4B59711: __libc_freeres (in /usr/lib/x86_64-linux-gnu/libc-2.29.so)
==70== by 0x482E19E: _vgnU_freeres (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_core-amd64-linux.so)
==70== by 0x4A0A3A9: __run_exit_handlers (exit.c:132)
==70== by 0x4A0A3D9: exit (exit.c:139)
==70== by 0x49E9B71: (below main) (libc-start.c:342)
==70== Block was alloc'd at
==70== at 0x4838DBF: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==70== by 0x303D6D: boost::detail::make_external_thread_data() (in /build-context/bin/debug/setmatch-tests)
==70== by 0x305424: boost::detail::add_new_tss_node(void const*, boost::shared_ptr<boost::detail::tss_cleanup_function>, void*) (in /build-context/bin/debug/setmatch-tests)
==70== by 0x3054ED: boost::detail::set_tss_data(void const*, boost::shared_ptr<boost::detail::tss_cleanup_function>, void*, bool) (in /build-context/bin/debug/setmatch-tests)
==70== by 0x188841: boost::thread_specific_ptr<burningmime::setmatch::MatchState>::reset(burningmime::setmatch::MatchState*) (tss.hpp:105)
[...]
==70== 24 bytes in 1 blocks are definitely lost in loss record 1 of 2
==70== at 0x4838DBF: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==70== by 0x303F50: boost::detail::make_external_thread_data() (in /build-context/bin/debug/setmatch-tests)
==70== by 0x305424: boost::detail::add_new_tss_node(void const*, boost::shared_ptr<boost::detail::tss_cleanup_function>, void*) (in /build-context/bin/debug/setmatch-tests)
==70== by 0x3054ED: boost::detail::set_tss_data(void const*, boost::shared_ptr<boost::detail::tss_cleanup_function>, void*, bool) (in /build-context/bin/debug/setmatch-tests)
[...]
It looks like boost is allocating its thread data in the same place that dlerror has allocated its own thread data. A quick search points to a (slightly different?) version of dlerror here
A quick glance over at bosot's code looks to me like it's just allocating the TSS block on the heap.
This was not a problem with GCC 7.3.0 + Ubuntu 18 (same Boost version)
Anyone have some insight here?
EDIT: Maybe it's the double-free that was fixed in this commit? Still I don't see why Boost would be using that at all.
Please check the version of all the tools you had use. It seems like there is some version compatibility issue in this. Try using 3.15.0 version of valgrind.
See here for the usage of valgrind.
If I modify the glibc upstream test case around the pthread_setspecific call like this (and compile it with g++):
void *ptr = new char;
printf("Setting thread local to ptr.\n");
if (pthread_setspecific(key, ptr) != 0) {
perror("pthread_setspecific");
exit(1);
}
delete ptr;
I get this error when running against glibc from right before the fix (at commit 5b06f538c5aee0389ed034f60d90a8884d6d54de, using ./testrun.sh --tool=valgrind /path/to/test from the glibc build tree):
==14143== Invalid read of size 8
==14143== at 0x483B550: check_free (dlerror.c:188)
==14143== by 0x483BA21: free_key_mem (dlerror.c:221)
==14143== by 0x483BA21: __dlerror_main_freeres (dlerror.c:239)
==14143== by 0x4D06AD1: __libc_freeres (in /home/fweimer/src/gnu/glibc/build/libc.so)
==14143== by 0x48031DE: _vgnU_freeres (vg_preloaded.c:77)
==14143== by 0x4BDD331: __run_exit_handlers (exit.c:132)
==14143== by 0x4BDD3C9: exit (exit.c:139)
==14143== by 0x4BC7E21: (below main) (libc-start.c:342)
==14143== Address 0x4d750d8 is 23 bytes after a block of size 1 free'd
==14143== at 0x480CEFC: operator delete(void*) (vg_replace_malloc.c:586)
==14143== by 0x401344: main (t.c:93)
==14143== Block was alloc'd at
==14143== at 0x480BE86: operator new(unsigned long) (vg_replace_malloc.c:344)
==14143== by 0x4012F4: main (t.c:87)
==14143==
==14143== Invalid free() / delete / delete[] / realloc()
==14143== at 0x480CA0C: free (vg_replace_malloc.c:540)
==14143== by 0x483BA29: free_key_mem (dlerror.c:223)
==14143== by 0x483BA29: __dlerror_main_freeres (dlerror.c:239)
==14143== by 0x4D06AD1: __libc_freeres (in /home/fweimer/src/gnu/glibc/build/libc.so)
==14143== by 0x48031DE: _vgnU_freeres (vg_preloaded.c:77)
==14143== by 0x4BDD331: __run_exit_handlers (exit.c:132)
==14143== by 0x4BDD3C9: exit (exit.c:139)
==14143== by 0x4BC7E21: (below main) (libc-start.c:342)
==14143== Address 0x4d750c0 is 0 bytes inside a block of size 1 free'd
==14143== at 0x480CEFC: operator delete(void*) (vg_replace_malloc.c:586)
==14143== by 0x401344: main (t.c:93)
==14143== Block was alloc'd at
==14143== at 0x480BE86: operator new(unsigned long) (vg_replace_malloc.c:344)
==14143== by 0x4012F4: main (t.c:87)
This is pretty much the same error that you got, minus the nesting of the operator new allocation within Boost. So it looks indeed like the two bugs are the same.
This makes sense: Due to bug 24476, libdl uses an uninitialized pthread_key_t value (without previously calling pthread_key_create on it). For the data segment (where internal key for libdl is stored0, uninitialized means zero, of course, and as you can see from the diagnostic output in the test, the key allocated by the test (and Boost in your case) was in fact key 0:
key = 0
This libdl code is rather convoluted, and I posted a patch which moves dlerror into libc (from libdl) and also avoids using POSIX threads thread-local storage altogether.
To summarize: Whoever maintains the glibc version you use needs to backport the upstream fix into their source tree and release an update. We had to do this as well. On the plus side, this bug only happens when you run your application under valgrind and similar tools because during regular process shutdown, __libc_freeres is not invoked: the process will exit soon anyway, and the kernel cleans up all the resources for us. Unless you use valgrind in production, this means that you will never encounter this bug there. Of course, it's still an annoying issue when you are using valgrind for debugging. Sorry about that.
Perhaps you should upgrade the valgrind version to 3.15.0 , It should help.
i think here should help you.
I am executing simple tensorflow code to create graph def as shown
tensorflow::NewSession (options, &session)
ReadBinaryProto (tensorflow::Env::Default(), "/home/ashok/eclipseWorkspace/faceRecognition-x86_64/Data/models/optimized_facenet.pb", &graph_def));
session->Create (graph_def);
But when I run Valgrind as shown below
valgrind --leak-check=full --show-leak-kinds=all --vex-guest-max-insns=25 ./faceRecognition-x86_64 -r -i
I get below errors
==12366== 16,000 bytes in 1 blocks are still reachable in loss record 47,782 of 47,905
==12366== at 0x4C2E19F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12366== by 0xBF875DC: std::vector<tensorflow::CostModel::MemUsage, std::allocator<tensorflow::CostModel::MemUsage> >::reserve(unsigned long) (in /usr/lib/libtensorflow_cc.so)
==12366== by 0xBF90128: tensorflow::CostModel::InitFromGraph(tensorflow::Graph const&) (in /usr/lib/libtensorflow_cc.so)
==12366== by 0xBEE48D3: tensorflow::SimpleGraphExecutionState::InitBaseGraph(tensorflow::BuildGraphOptions const&) (in /usr/lib/libtensorflow_cc.so)
==12366== by 0xBEE52CF: tensorflow::SimpleGraphExecutionState::MakeForBaseGraph(tensorflow::GraphDef*, tensorflow::SimpleGraphExecutionStateOptions const&, std::unique_ptr<tensorflow::SimpleGraphExecutionState, std::default_delete<tensorflow::SimpleGraphExecutionState> >*) (in /usr/lib/libtensorflow_cc.so)
==12366== by 0xBE68B9D: tensorflow::DirectSession::MaybeInitializeExecutionState(tensorflow::GraphDef const&, bool*) (in /usr/lib/libtensorflow_cc.so)
==12366== by 0xBE68CF9: tensorflow::DirectSession::ExtendLocked(tensorflow::GraphDef const&) (in /usr/lib/libtensorflow_cc.so)
==12366== by 0xBE68FC7: tensorflow::DirectSession::Create(tensorflow::GraphDef const&) (in /usr/lib/libtensorflow_cc.so)
==12366== by 0x26B899: TensorFlow::initializeRecognition() (in /home/ashok/eclipseWorkspace/faceRecognition-x86_64/Debug/faceRecognition-x86_64)
==12366== by 0x24197D: RecognitionWithImages::RecognitionWithImages() (in /home/ashok/eclipseWorkspace/faceRecognition-x86_64/Debug/faceRecognition-x86_64)
==12366== by 0x12F27C: main (in /home/ashok/eclipseWorkspace/faceRecognition-x86_64/Debug/faceRecognition-x86_64)
These type of errors are also generated when I do session -> run ()
Due to the above issues, the memory needed to run the program keeps increasing as time passes and the application crashes due to insufficient memory after a certain point of time
Did you close the session ? You also need to delete the Session pointer.
session->Close();
delete session;
I'm seeing a truly baffling series of error reports from Valgrind's Memcheck tool:
==29456== Invalid read of size 8
==29456== at 0x4D5C90: CkIndex_Ping1::_callthr_trecv_PingMsg(CkThrCallArg*) (in /scratch/phil/charm/net-linux-x86_64-bigsim/tests/charm++/pingpong/pgm)
==29456== by 0x503ECB: CthStartThread (libthreads-default.c:1690)
==29456== by 0x56A08AF: ??? (in /lib/x86_64-linux-gnu/libc-2.19.so)
==29456== Address 0x5b09a90 is 0 bytes inside a block of size 16 alloc'd
==29456== at 0x4C2B0E0: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==29456== by 0x4D5C14: CkIndex_Ping1::_call_trecv_PingMsg(void*, void*) (in /scratch/phil/charm/net-linux-x86_64-bigsim/tests/charm++/pingpong/pgm)
==29456== by 0x517D79: CkDeliverMessageFree (ck.C:593)
==29456== by 0x5378C3: CkLocRec_local::invokeEntry(CkMigratable*, void*, int, bool) (cklocation.C:1795)
==29456== by 0x537CA7: CkLocRec_local::deliver(CkArrayMessage*, CkDeliver_t, int) (cklocation.C:1862)
==29456== by 0x539977: CkLocMgr::deliver(CkMessage*, CkDeliver_t, int) (cklocation.C:2834)
==29456== by 0x51F091: CkLocMgr::deliverInline(CkMessage*) (cklocation.h:313)
==29456== by 0x51A6EF: _processArrayEltMsg(CkCoreState*, envelope*) (ck.C:1181)
==29456== by 0x51A8C8: _processHandler(void*, CkCoreState*) (ck.C:1266)
==29456== by 0x4EE7EF: BgProcessMessageDefault(threadInfo*, char*) (blue.C:1339)
==29456== by 0x5C5928: BgProcessMessageFreezeMode(threadInfo*, char*) (middle-ccs.C:165)
==29456== by 0x4F590D: workThreadInfo::scheduler(int) (bigsim_proc.C:282)
Note that it's saying that the offending address is inside a still-allocated (i.e. not yet free()'d) block, and that the read size plus offset is well less than the size of the block.
This is on Ubuntu Linux 14.04, with Valgrind version valgrind-3.10.0.SVN (package 1:3.10~20140411-0ubuntu1), and the code was compiled with gcc/g++ 4.8.4-2ubuntu1~14.04.
I've found a similar question, to which the answer was "this is a bug on Mac OS X". Am I really looking at a Valgrind bug here, or is there something else my code might have wrong?
Edit: I also found a mailing list post covering a similar environment - user-level threads that might be screwing with Valgrind's understanding. It doesn't seem to actually answer anything though.
I have been digging several days in my 1500 lines of code to find those 15 bytes (possibly lost) to no avail.
There are no sufficient data provided by valgrind even though I ran the following command:
valgrind --leak-check=full --show-reachable=yes --track-origins=yes --show-below-main=yes ./myapp
I get the following block of report:
==3283== 15 bytes in 1 blocks are possibly lost in loss record 1 of 4
==3283== at 0x402842F: operator new(unsigned int) (vg_replace_malloc.c:255)
==3283== by 0x40D2A83: std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) (in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16)
==3283== by 0x40D4CF7: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16)
==3283== by 0x40D4E65: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (in /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16)
==3283== by 0x804DB22: _GLOBAL__sub_I__ZN7processC2Ei7in_addr (main.cpp:1304)
==3283== by 0x8050131: __libc_csu_init (in /home/username/myapp-write/src/myapp)
==3283== by 0x41A60A9: __libc_start_main (libc-start.c:185)
==3283== by 0x80499C0: ??? (in /home/username/myapp-write/src/myapp)
==3283==
Would any one, please, tell me how to detect the faulty line?
If your code invokes alternative termination strategies that are not designed to properly clean up automatic variables due to hindering unwind semantics, your automatic variable destructors will not be called.
std::terminate, as you mentioned in-comment you were using, unfortunately ponies up one such condition. The default action for the termination handler is to invoke std::abort, which does not fire cleanup destruction on automatic, thread-local, or static storage duration objects, and any such vars that have assumed dynamic memory management will leak like a sieve.
Avoid termination in this fashion unless you have a very good reason for it, and in general there are very few good reasons for it.
Best of luck.