Related
When I test run C++ programs with the option -fsanitize=address I see many locations point to an indirect leak. They are all initiated from libasan.so. The exact error shown below. I am not sure that I am responsible for this leak or not. I was able to fixed other direct leaks in my program.
Indirect leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x7f24e2b9c947 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
#1 0x7f24e2517dc8 in __gnu_cxx::new_allocator<std::pair<char, int> >::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
#2 0x7f24e250ed74 in std::allocator_traits<std::allocator<std::pair<char, int> > >::allocate(std::allocator<std::pair<char, int> >&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:444
#3 0x7f24e24ff853 in std::_Vector_base<std::pair<char, int>, std::allocator<std::pair<char, int> > >::_M_allocate(unsigned long) /usr/include/c++/9/bits/stl_vector.h:343
#4 0x7f24e24e8aa6 in std::_Vector_base<std::pair<char, int>, std::allocator<std::pair<char, int> > >::_M_create_storage(unsigned long) /usr/include/c++/9/bits/stl_vector.h:358
#5 0x7f24e24d8ee8 in std::_Vector_base<std::pair<char, int>, std::allocator<std::pair<char, int> > >::_Vector_base(unsigned long, std::allocator<std::pair<char, int> > const&) /usr/include/c++/9/bits/stl_vector.h:302
#6 0x7f24e24c9782 in std::vector<std::pair<char, int>, std::allocator<std::pair<char, int> > >::vector(std::vector<std::pair<char, int>, std::allocator<std::pair<char, int> > > const&) /usr/include/c++/9/bits/stl_vector.h:552
#7 0x7f24e2579da6 in Hitbrief::Hitbrief(Hitbrief const&) /home/kzhou/coding/molbar/src/readcompass.h:21
#8 0x7f24e2579e5b in std::pair<Hitbrief const, int>::pair<Hitbrief&&, 0ul>(std::tuple<Hitbrief&&>&, std::tuple<>&, std::_Index_tuple<0ul>, std::_Index_tuple<>) /usr/include/c++/9/tuple:1674
#9 0x7f24e25798f0 in std::pair<Hitbrief const, int>::pair<Hitbrief&&>(std::piecewise_construct_t, std::tuple<Hitbrief&&>, std::tuple<>) /usr/include/c++/9/tuple:1663
#10 0x7f24e2578f77 in void __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<Hitbrief const, int> > >::construct<std::pair<Hitbrief const, int>, std::piecewise_construct_t const&, std::tuple<Hitbrief&&>, std::tuple<> >(std::pair<Hitbrief const, int>*, std::piecewise_construct_t const&, std::tuple<Hitbrief&&>&&, std::tuple<>&&) /usr/include/c++/9/ext/new_allocator.h:147
#11 0x7f24e2577db5 in void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<Hitbrief const, int> > > >::construct<std::pair<Hitbrief const, int>, std::piecewise_construct_t const&, std::tuple<Hitbrief&&>, std::tuple<> >(std::allocator<std::_Rb_tree_node<std::pair<Hitbrief const, int> > >&, std::pair<Hitbrief const, int>*, std::piecewise_construct_t const&, std::tuple<Hitbrief&&>&&, std::tuple<>&&) /usr/include/c++/9/bits/alloc_traits.h:484
#12 0x7f24e2575199 in void std::_Rb_tree<Hitbrief, std::pair<Hitbrief const, int>, std::_Select1st<std::pair<Hitbrief const, int> >, std::less<Hitbrief>, std::allocator<std::pair<Hitbrief const, int> > >::_M_construct_node<std::piecewise_construct_t const&, std::tuple<Hitbrief&&>, std::tuple<> >(std::_Rb_tree_node<std::pair<Hitbrief const, int> >*, std::piecewise_construct_t const&, std::tuple<Hitbrief&&>&&, std::tuple<>&&) /usr/include/c++/9/bits/stl_tree.h:614
#13 0x7f24e25714e9 in std::_Rb_tree_node<std::pair<Hitbrief const, int> >* std::_Rb_tree<Hitbrief, std::pair<Hitbrief const, int>, std::_Select1st<std::pair<Hitbrief const, int> >, std::less<Hitbrief>, std::allocator<std::pair<Hitbrief const, int> > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<Hitbrief&&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<Hitbrief&&>&&, std::tuple<>&&) /usr/include/c++/9/bits/stl_tree.h:631
#14 0x7f24e256e7ae in std::_Rb_tree_iterator<std::pair<Hitbrief const, int> > std::_Rb_tree<Hitbrief, std::pair<Hitbrief const, int>, std::_Select1st<std::pair<Hitbrief const, int> >, std::less<Hitbrief>, std::allocator<std::pair<Hitbrief const, int> > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<Hitbrief&&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<Hitbrief const, int> >, std::piecewise_construct_t const&, std::tuple<Hitbrief&&>&&, std::tuple<>&&) /usr/include/c++/9/bits/stl_tree.h:2455
#15 0x7f24e2568f82 in std::map<Hitbrief, int, std::less<Hitbrief>, std::allocator<std::pair<Hitbrief const, int> > >::operator[](Hitbrief&&) /usr/include/c++/9/bits/stl_map.h:518
Even when all leaks are indirect, there is still memory leak in the source code. I was able to track down all leak and eliminate all indirect leaks. Thank for everyone feedback. In the past, eliminating direct leaks removed all the indirect leaks. This is my first experience, I think it is good to share with someone who experienced this situation for the first time.
Consider the following code snippet:
myEnum stringToEnum(const std::string& enumString) {
static const std::unordered_map<std::string, myEnum> conversionMap = {
{"enumA", myEnum::enumA},
{"enumB", myEnum::enumB},
{"enumC", myEnum::enumC}};
if(conversionMap.count(enumString) == 0) {
// Return some default value
}
return conversionMap.at(enumString);
}
void threadedFcn() {
// Do some things
for (int i = 0; i < someNumber; ++i) {
// Do some more things
auto myEnum = stringToEnum(myString);
// Do even more stuff
}
}
int main() {
threadedFcn(); // No problem
std::thread(threadedFcn).join(); // No problem
std::thread(threadedFcn).detach(); // Seg fault
}
A few things about this:
If I remove the call to count then it won't seg fault. It looks like it's caused by that somehow.
It only happens on Linux Debian 9 (I've built with Windows 10 and Mac as well).
If I add something like std::this_thread::sleep_for(std::chrono::milliseconds(2000)); after the detach call, then it won't seg fault.
If I make conversionMap not static, then it won't seg fault.
I can't figure out what the exact problem is, but it's something to do with the main thread exiting before the detached thread finishes.
As Slava correctly said, the problem is that you are letting the main return, while your other thread is running.
Returning from main eventually calls exit, which goes through all registered atexit handlers.
When using libstdc++ (and probably most other C++ runtime implementations), any static C++ object that has been constructed registers its destructor with atexit, so after that destructor fires, your detached thread is accessing destroyed conversionMap object, with predictable results.
You can observe this by using Address Sanitizer (-fsanitize=address), which reports:
=================================================================
==87625==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000000010 at pc 0x563122b9c6c5 bp 0x7fc54eafea20 sp 0x7fc54eafea18
READ of size 8 at 0x603000000010 thread T2
#0 0x563122b9c6c4 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> >, std::__detail
::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>
>::_M_bucket_begin(unsigned long) const /usr/include/c++/9/bits/hashtable.h:943
#1 0x563122b9bcf2 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> >, std::__detail
::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>
>::count(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /usr/include/c++/9/bits/hashtable.h:1451
#2 0x563122b9b76e in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, myEnum, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::ba
sic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> > >::count(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /usr/include/c++/9/bits/unordered_map.h:939
#3 0x563122b9a793 in stringToEnum(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /tmp/t.cc:16
#4 0x563122b9aa9c in threadedFcn() /tmp/t.cc:27
#5 0x563122b9f71e in void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) /usr/include/c++/9/bits/invoke.h:60
#6 0x563122b9f681 in std::__invoke_result<void (*)()>::type std::__invoke<void (*)()>(void (*&&)()) /usr/include/c++/9/bits/invoke.h:95
#7 0x563122b9f5cb in void std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/include/c++/9/thread:244
#8 0x563122b9f56c in std::thread::_Invoker<std::tuple<void (*)()> >::operator()() /usr/include/c++/9/thread:251
#9 0x563122b9f4ed in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run() /usr/include/c++/9/thread:195
#10 0x7fc552301baf (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xcebaf)
#11 0x7fc55202bf26 in start_thread /build/glibc-M65Gwz/glibc-2.30/nptl/pthread_create.c:479
#12 0x7fc5521532ee in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xfd2ee)
0x603000000010 is located 0 bytes inside of 24-byte region [0x603000000010,0x603000000028)
freed by thread T0 here:
#0 0x7fc552509f97 in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x109f97)
#1 0x563122b9e5fb in __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::deallocate(std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128
#2 0x563122b9dbf5 in std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::deallocate(std::allocator<std::__detail::_Hash_node_base*>&, std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:470
#3 0x563122b9d0cc in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum>, true> > >::_M_deallocate_buckets(std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/9/bits/hashtable_policy.h:2148
#4 0x563122b9c5d3 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> >, std::__detail
::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>
>::_M_deallocate_buckets(std::__detail::_Hash_node_base**, unsigned long) /usr/include/c++/9/bits/hashtable.h:370
#5 0x563122b9bc99 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> >, std::__detail
::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>
>::_M_deallocate_buckets() /usr/include/c++/9/bits/hashtable.h:375
#6 0x563122b9b73b in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> >, std::__detail
::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>
>::~_Hashtable() /usr/include/c++/9/bits/hashtable.h:1353
#7 0x563122b9b4ff in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, myEnum, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::ba
sic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> > >::~unordered_map() /usr/include/c++/9/bits/unordered_map.h:102
#8 0x7fc552093e26 in __run_exit_handlers /build/glibc-M65Gwz/glibc-2.30/stdlib/exit.c:108
previously allocated by thread T0 here:
#0 0x7fc55250919f in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10919f)
#1 0x563122b9e828 in __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::allocate(unsigned long, void const*) /usr/include/c++/9/ext/new_allocator.h:114
#2 0x563122b9defc in std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::allocate(std::allocator<std::__detail::_Hash_node_base*>&, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:444
#3 0x563122b9d6d4 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum>, true> > >::_M_allocate_buckets(unsigned long) /usr/include/c++/9/bits/hashtable_policy.h:2134
#4 0x563122b9ce5b in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> >, std::__detail
::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>
>::_M_allocate_buckets(unsigned long) /usr/include/c++/9/bits/hashtable.h:361
#5 0x563122b9c3db in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> >, std::__detail
::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>
>::_Hashtable<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> const*>(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> const*, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> const*, unsigned l
ong, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::__detail::_Mod_range_hashing const&, std::__detail::_Default_ranged_hash const&, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::__detail::_Select1st const&, std::allocator<std::pair<std::__cxx
11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> > const&) /usr/include/c++/9/bits/hashtable.h:989
#6 0x563122b9bab5 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> >, std::__detail
::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>
>::_Hashtable(std::initializer_list<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> >, unsigned long, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&,
std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> > const&) /usr/include/c++/9/bits/hashtable.h:466
#7 0x563122b9b6a3 in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, myEnum, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::ba
sic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> > >::unordered_map(std::initializer_list<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> >, unsigned long, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::equal_to<std::__
cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, myEnum> > const&) /usr/include/c++/9/bits/unordered_map.h:231
#8 0x563122b9a678 in stringToEnum(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /tmp/t.cc:14
#9 0x563122b9aa9c in threadedFcn() /tmp/t.cc:27
#10 0x563122b9ac0f in main /tmp/t.cc:33
#11 0x7fc55207ce0a in __libc_start_main ../csu/libc-start.c:308
Thread T2 created by T0 here:
#0 0x7fc5524399b2 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x399b2)
#1 0x7fc552301e24 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xcee24)
#2 0x563122b9ac75 in main /tmp/t.cc:35
#3 0x7fc55207ce0a in __libc_start_main ../csu/libc-start.c:308
Update:
I guess an easy fix is to just make my variable not-static (functionally it will be the same but with an obvious performance hit).
There is an easy way to avoid having to construct a new unordered_map on every call, while also avoiding the global destruction problem: use a static pointer:
myEnum stringToEnum(const std::string& enumString) {
static const auto *conversionMap =
new std::unordered_map<std::string, myEnum>{
{"enumA", myEnum::enumA},
{"enumB", myEnum::enumB},
{"enumC", myEnum::enumC}
};
if (conversionMap->count(enumString) == 0) {
// Return some default value
}
return conversionMap->at(enumString);
}
I used valgrind to check my project written in C++. It told me there was a definitely memory leak in a line with vector assignment.
What valgrind told is
==37533== 514,604 bytes in 124,984 blocks are definitely lost in loss record 369 of 433
==37533== at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==37533== by 0x42BB89: __gnu_cxx::new_allocator<int>::allocate(unsigned long, void const*) (new_allocator.h:104)
==37533== by 0x42832B: std::allocator_traits<std::allocator<int> >::allocate(std::allocator<int>&, unsigned long) (alloc_traits.h:491)
==37533== by 0x424565: std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) (stl_vector.h:170)
==37533== by 0x48B73F: int* std::vector<int, std::allocator<int> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > > >(unsigned long, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >) (stl_vector.h:1224)
==37533== by 0x48B2BD: std::vector<int, std::allocator<int> >::operator=(std::vector<int, std::allocator<int> > const&) (vector.tcc:195)
==37533== by 0x603A49: Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::buildTrie(std::vector<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TermCode, std::allocator<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TermCode> > const&, std::vector<std::vector<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TrieNode, std::allocator<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TrieNode> >*, std::allocator<std::vector<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TrieNode, std::allocator<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TrieNode> >*> >&) (Datrie.cpp:467)
==37533== by 0x602758: Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::build(std::vector<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TermCode, std::allocator<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TermCode> > const&) (Datrie.cpp:178)
==37533== by 0x602103: Loader::Trie_Loader::InnerLoad(BaseDict_Collection&) (Trie_Loader.cpp:116)
==37533== by 0x473BA6: deduce::DeduceClassify::Get_Dat_EntityRec() (DeduceClassify.cpp:500)
==37533== by 0x470D7A: deduce::DeduceClassify::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Qp_Dict> const&, Analyse::Analysis*, MINISEARCH::Index*, Normalization::Normalizer*) (DeduceClassify.cpp:53)
==37533== by 0x40E784: queryparser::CQueryParserImpl::Init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (QueryParserImpl.cpp:120)
The code snippet is
if ((int) (*itVecIter)->code.size() == nLevel + 1)
{
treeNode.bEnd = true;
// NOTE: the memory leak happens at following
treeNode.weight = (*itVecIter)->weight;
itVecIter = vecIter.erase(itVecIter);
}
And the runtime callstack via gdb is
(gdb) bt
#0 Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::buildTrie (this=0x7fffffffd8f8, vkey_code=std::vector of length 125163, capacity 131072 = {...},
trie=std::vector of length 31422, capacity 31422 = {...}) at ./tools/Dict/Datrie.cpp:467
#1 0x0000000000602759 in Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::build (this=0x7fffffffd8f8, vkey_code=std::vector of length 125163, capacity 131072 = {...})
at ./tools/Dict/Datrie.cpp:178
#2 0x0000000000602104 in Loader::Trie_Loader::InnerLoad (this=0x7fffffffd8e0, dictPtrs=...) at loader//Trie_Loader.cpp:116
#3 0x0000000000473ba7 in deduce::DeduceClassify::Get_Dat_EntityRec (this=0xa2fb28) at Classification/Deduce//DeduceClassify.cpp:500
#4 0x0000000000470d7b in deduce::DeduceClassify::init (this=0xa2fb28, base_Path="..//data/",
qp_dict=std::shared_ptr (count 2, weak 0) 0x1bc14600, tokenizer=0xa2f998, indexer=0xf6efb0, pNormer=0xa2f898)
at Classification/Deduce//DeduceClassify.cpp:53
#5 0x000000000040e785 in queryparser::CQueryParserImpl::Init (this=0xa2f4a0, dataPath="../") at QueryParserImpl.cpp:120
#6 0x000000000043ca92 in main_process::test_main (argc=2, argv=0x7fffffffdd20) at MainProcess.cpp:284
#7 0x00000000006301ae in main (argc=3, argv=0x7fffffffde38) at main.cpp:57
I used template class in Datrie.cpp. And I am not sure why vector assignment with
treeNode.weight = (*itVecIter)->weight;
has memory leaks.
Thanks.
It seems that the assignment operator of treeNode.weight is called, it allocates some memory to use but never de-allocated.
==37533== by 0x48B73F: int* std::vector<int, std::allocator<int> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > > >(unsigned long, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >) (stl_vector.h:1224)
==37533== by 0x48B2BD: std::vector<int, std::allocator<int> >::operator=(std::vector<int, std::allocator<int> > const&) (vector.tcc:195)
As #n.m.'s suggestion, I've checked the code whole around. And I think I found the issue. Simplified the class constructor/destructor and manipulation in following:
#include <iostream>
#include <vector>
#include <string.h>
using namespace std;
struct leak_struct {
vector<string> array;
int set;
};
int main(){
leak_struct ls;
ls.array.push_back("hello");
ls.array.push_back("world");
memset(&ls, 0, sizeof(ls));
ls.array.push_back("code");
ls.array.push_back("review");
memset(&ls, 0, sizeof(ls));
return 0;
}
The reason it leaks is at the line where memset is called, this call will nullify the vector pointer in struct and makes the new allocated buffer not freed.
Recently other people upgraded our boost library, now I see this coredump when a read_json function is called heavily. Never seen this before. And I have #define BOOST_SPIRIT_THREADSAFE in the code, that's why this didn't happen before. Really need help!Thanks!
Program terminated with signal 11, Segmentation fault.
#0 0x00007f062ce0d094 in std::mem_fun1_t<int, boost::spirit::classic::impl::grammar_helper_base<boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> > >, boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> >*>::operator() (this=0x7f06120f4c90, __p=0x7f05f813faa0, __x=0x7f06120f4df0) at /usr/lib/gcc/x86_64-spcdn-linux-gnu/4.5.2/../../../../include/c++/4.5.2/bits/stl_function.h:605
in /usr/lib/gcc/x86_64-spcdn-linux-gnu/4.5.2/../../../../include/c++/4.5.2/bits/stl_function.h
#0 0x00007f062ce0d094 in std::mem_fun1_t<int, boost::spirit::classic::impl::grammar_helper_base<boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> > >, boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> >*>::operator() (this=0x7f06120f4c90, __p=0x7f05f813faa0, __x=0x7f06120f4df0) at /usr/lib/gcc/x86_64-spcdn-linux-gnu/4.5.2/../../../../include/c++/4.5.2/bits/stl_function.h:605
#1 0x00007f062ce0c66e in std::binder2nd<std::mem_fun1_t<int, boost::spirit::classic::impl::grammar_helper_base<boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> > >, boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> >*> >::operator() (this=0x7f06120f4c90, __x=#0x7f06001446e0) at /usr/lib/gcc/x86_64-spcdn-linux-gnu/4.5.2/../../../../include/c++/4.5.2/backward/binders.h:154
#2 0x00007f062ce0b87e in std::for_each<std::reverse_iterator<__gnu_cxx::__normal_iterator<boost::spirit::classic::impl::grammar_helper_base<boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> > >**, std::vector<boost::spirit::classic::impl::grammar_helper_base<boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> > >*, std::allocator<boost::spirit::classic::impl::grammar_helper_base<boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> > >*> > > >, std::binder2nd<std::mem_fun1_t<int, boost::spirit::classic::impl::grammar_helper_base<boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> > >, boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> >*> > > (__first=..., __last=..., __f=...) at /usr/lib/gcc/x86_64-spcdn-linux-gnu/4.5.2/../../../../include/c++/4.5.2/bits/stl_algo.h:4185
#3 0x00007f062ce0aa88 in boost::spirit::classic::impl::grammar_destruct<boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> > > (self=0x7f06120f4df0) at /usr/include/boost/spirit/home/classic/core/non_terminal/impl/grammar.ipp:324
#4 0x00007f062ce09c5f in boost::spirit::classic::grammar<boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, boost::spirit::classic::parser_context<boost::spirit::classic::nil_t> >::~grammar (this=0x7f06120f4df0, __in_chrg=<optimized out>) at /usr/include/boost/spirit/home/classic/core/non_terminal/grammar.hpp:52
#5 0x00007f062ce093f7 in boost::property_tree::json_parser::json_grammar<boost::property_tree::basic_ptree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::~json_grammar (this=0x7f06120f4df0, __in_chrg=<optimized out>) at /usr/include/boost/property_tree/detail/json_parser_read.hpp:162
#6 0x00007f062ce0987f in boost::property_tree::json_parser::read_json_internal<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > > (stream=..., pt=..., filename="") at /usr/include/boost/property_tree/detail/json_parser_read.hpp:310
#7 0x00007f062ce08eb6 in boost::property_tree::json_parser::read_json<boost::property_tree::basic_ptree<std::string, std::string, std::less<std::string> > > (stream=..., pt=...) at /usr/include/boost/property_tree/json_parser.hpp:45
Everywhere you include json_parser.hpp or Spirit, BOOST_SPIRIT_THREADSAFE must be defined. Defining it globally would be safest, in case another part of boost/yourApp uses Spirit without your knowledge.
For any inclusion where this isn't defined, Sprit annoyingly makes a static variable in grammar.ipp that'll be shared amongst all instances of Spirit. Your grammar_destruct error is because Spirit is missing this define in one of your modules.
This question already has answers here:
Is it smart to replace boost::thread and boost::mutex with c++11 equivalents?
(7 answers)
Closed 9 years ago.
have a code which used to run fine with boost thread under ubuntu. it's basic read only data sharing multithreading. I try to use C++11 instead of boost, very basic transition. the code compiles but have subtle bugs. crashes randomly with C++11 std thread. tried to use valgrind drd, but hard to read the debug info. Any thoughts?
==19608== Conflicting load by thread 3 at 0x00643be8 size 8
==19608== at 0x41CEBA: std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul>::operator()() (random.tcc:601)
==19608== by 0x41CD6C: double std::generate_canonical<double, 53ul, std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul> >(std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul>&) (random.tcc:3475)
==19608== by 0x41CB06: std::__detail::_Adaptor<std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul>, double>::operator()() (random.h:190)
==19608== by 0x41C877: double std::normal_distribution<double>::operator()<std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul> >(std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul>&, std::normal_distribution<double>::param_type const&) (random.tcc:1950)
==19608== by 0x41C688: double std::normal_distribution<double>::operator()<std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul> >(std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul>&) (random.h:2196)
==19608== by 0x41C379: nrand(double, double) (NRand.cpp:8)
==19608== by 0x416F78: ClassDef::set_x() (LoanDef.h:407)
==19608== by 0x4168CA: Sim(std::vector<ClassDef, std::allocator<ClassDef> >&, Assumption&, std::unordered_map<std::string, std::vector<double, std::allocator<double> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<double, std::allocator<double> > > > >&, unsigned int, NextStepCalcML&, std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >&, unsigned int, unsigned int) (NewSim.cpp:105)
==19608== by 0x411A09: _ZNSt12_Bind_simpleIFPFvRSt6vectorI7LoanDefSaIS1_EER10AssumptionRSt13unordered_mapISsS0_IdSaIdEESt4hashISsESt8equal_toISsESaISt4pairIKSsS9_EEEjR14NextStepCalcMLRS0_IS0_IS9_SaIS9_EESaISN_EEjjESt17reference_wrapperIS3_EST_IS5_EST_ISI_EjST_ISK_EST_ISP_EjjEE9_M_invokeIILm0ELm1ELm2ELm3ELm4ELm5ELm6ELm7EEEEvSt12_Index_tupleIIXspT_EEE (functional:1732)
==19608== by 0x4116AE: std::_Bind_simple<void (*()(std::reference_wrapper<std::vector<LoanDef, std::allocator<LoanDef> > >, std::reference_wrapper<Assumption>, std::reference_wrapper<std::unordered_map<std::string, std::vector<double, std::allocator<double> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<double, std::allocator<double> > > > > >, unsigned int, std::reference_wrapper<NextStepCalcML>, std::reference_wrapper<std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > > >, unsigned int, unsigned int))(std::vector<LoanDef, std::allocator<ClassDef> >&, Assumption&, std::unordered_map<std::string, std::vector<double, std::allocator<double> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<double, std::allocator<double> > > > >&, unsigned int, NextStep&, std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >&, unsigned int, unsigned int)>::operator()() (functional:1720)
==19608== by 0x411647: std::thread::_Impl<std::_Bind_simple<void (*()(std::reference_wrapper<std::vector<ClassDef, std::allocator<ClassDef> > >, std::reference_wrapper<Assumption>, std::reference_wrapper<std::unordered_map<std::string, std::vector<double, std::allocator<double> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<double, std::allocator<double> > > > > >, unsigned int, std::reference_wrapper<NextStep>, std::reference_wrapper<std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > > >, unsigned int, unsigned int))(std::vector<LoanDef, std::allocator<LoanDef> >&, Assumption&, std::unordered_map<std::string, std::vector<double, std::allocator<double> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<double, std::allocator<double> > > > >&, unsigned int, NextStep&, std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >&, unsigned int, unsigned int)> >::_M_run() (thread:115)
thank you. btw, I tried to use this random head file I wrote, not sure if it's safe under multithread environment. it used to work fine with boost.
myrand.hpp
#ifndef NRAND_H
#define NRAND_H
#include <random>
double nrand(double mean = 0., double sd = 1.);
double urand(double a=0., double b=0.);
#endif
and myrand.cpp
#include "NRand.h"
using namespace std;
double nrand(double mean, double sd) {
static random_device rd;
static subtract_with_carry_engine<unsigned,24,10,24> e(rd());
normal_distribution<> dist(mean, sd);
return dist(e);
}
double urand(double a, double b) {
static random_device rd;
static subtract_with_carry_engine<unsigned,24,10,24> e(rd());
uniform_real_distribution<> dist(a, b);
return dist(e);
}
many thanks.
IMHO, your code is not threadsafe and hence should not work correctly under C++11. I think the problem is that the static variables rd and e are global variables but not protected (by mutex), so concurrent calls will race.
Presumably, you can make this code threadsafe by making those variables thread_local, but I have no experience.