I have written some C++ code that is structured as follows:
double kernel(params)
{
//code
}
void optimize(params)
{
//some code
double x = kernel();
//some more code
}
int main()
{
//some code
optimize();
//some more code
}
I tried profiling it with callgrind using the following commands:
g++ -O3 -g sgd.cpp
valgrind --tool=callgrind ./a.out commandline_args
callgrind_annotate callgrind.out.XXXX
I get the following output:
--------------------------------------------------------------------------------
Ir
--------------------------------------------------------------------------------
12,916,968,785 PROGRAM TOTALS
--------------------------------------------------------------------------------
Ir file:function
--------------------------------------------------------------------------------
5,862,783,191 /build/buildd/eglibc-2.15/string/../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:__memmove_ssse3 [/lib/i386-linux-gnu/libc-2.15.so]
2,847,653,393 /build/buildd/eglibc-2.15/malloc/malloc.c:_int_malloc [/lib/i386-linux-gnu/libc-2.15.so]
1,327,109,692 /build/buildd/eglibc-2.15/malloc/malloc.c:_int_free [/lib/i386-linux-gnu/libc-2.15.so]
847,560,182 sgd.cpp:main [a.out]
503,022,767 /build/buildd/eglibc-2.15/malloc/malloc.c:malloc [/lib/i386-linux-gnu/libc-2.15.so]
235,458,068 /build/buildd/eglibc-2.15/malloc/malloc.c:free [/lib/i386-linux-gnu/libc-2.15.so]
213,580,120 /build/buildd/eglibc-2.15/math/../sysdeps/i386/fpu/e_exp.S:__ieee754_exp [/lib/i386-linux-gnu/libm-2.15.so]
203,349,602 ???:operator new(unsigned int) [/usr/lib/i386-linux-gnu/libstdc++.so.6.0.16]
192,222,108 /build/buildd/eglibc-2.15/math/../sysdeps/ieee754/dbl-64/w_exp.c:exp [/lib/i386-linux-gnu/libm-2.15.so]
128,438,068 /build/buildd/eglibc-2.15/string/../sysdeps/i386/i686/multiarch/strcat.S:0x0012ac73 [/lib/i386-linux-gnu/libc-2.15.so]
128,431,176 ???:operator delete(void*) [/usr/lib/i386-linux-gnu/libstdc++.so.6.0.16]
128,358,564 /usr/include/c++/4.6/ext/new_allocator.h:main
117,645,255 /usr/include/c++/4.6/bits/stl_vector.h:main
112,167,083 /usr/include/c++/4.6/bits/stl_algobase.h:main
Except main(), it doesn't show which parts of the source code take up most of the time. I know for a fact that most of the time is spent in optimize() function, and in turn a significant fraction of that time in kernel() function, but I can't see this from the output.
How do I go about getting the details so that I can speed up my code?
If it helps, I am using std::vectors extensively in the code. I had implemented a similar code some time back using arrays, and callgrind seemed to work fine then. Could this be an issue?
If I disable the O3 flag, I get the following output:
--------------------------------------------------------------------------------
Ir
--------------------------------------------------------------------------------
19,026,610,083 PROGRAM TOTALS
--------------------------------------------------------------------------------
Ir file:function
--------------------------------------------------------------------------------
5,233,252,577 /build/buildd/eglibc-2.15/string/../sysdeps/i386/i686/multiarch/memcpy-ssse3.S:__memmove_ssse3 [/lib/i386-linux-gnu/libc-2.15.so]
2,542,000,057 /build/buildd/eglibc-2.15/malloc/malloc.c:_int_malloc [/lib/i386-linux-gnu/libc-2.15.so]
1,184,626,252 /build/buildd/eglibc-2.15/malloc/malloc.c:_int_free [/lib/i386-linux-gnu/libc-2.15.so]
983,472,430 sgd.cpp:optimize(std::vector<double, std::allocator<double> >, std::vector<int, std::allocator<int> >, std::vector<double, std::allocator<double> >) [a.out]
781,018,740 ???:std::vector<double, std::allocator<double> >::operator[](unsigned int) [a.out]
772,117,839 sgd.cpp:kernel(std::vector<double, std::allocator<double> >, int, int, double) [a.out]
476,616,742 ???:std::vector<double, std::allocator<double> >::vector(std::vector<double, std::allocator<double> > const&) [a.out]
449,016,969 /build/buildd/eglibc-2.15/malloc/malloc.c:malloc [/lib/i386-linux-gnu/libc-2.15.so]
324,200,916 ???:std::vector<double, std::allocator<double> >::size() const [a.out]
305,705,504 ???:std::_Vector_base<double, std::allocator<double> >::_Vector_base(unsigned int, std::allocator<double> const&) [a.out]
267,492,204 ???:std::_Vector_base<double, std::allocator<double> >::~_Vector_base() [a.out]
238,309,873 /usr/include/c++/4.6/bits/stl_algobase.h:double* std::__copy_move<false, true, std::random_access_iterator_tag>::__copy_m<double>(double const*, double const*, double*) [a.out]
238,308,370 /usr/include/c++/4.6/bits/stl_algobase.h:double* std::__copy_move_a2<false, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, double*>(__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, double*) [a.out]
228,776,040 /usr/include/c++/4.6/bits/stl_algobase.h:std::_Miter_base<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > > >::iterator_type std::__miter_base<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > > >(__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >) [a.out]
228,776,038 /usr/include/c++/4.6/bits/stl_algobase.h:double* std::copy<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, double*>(__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, double*) [a.out]
210,178,748 /build/buildd/eglibc-2.15/malloc/malloc.c:free [/lib/i386-linux-gnu/libc-2.15.so]
210,172,446 ???:std::vector<double, std::allocator<double> >::~vector() [a.out]
209,711,018 sgd.cpp:square(double) [a.out]
190,646,380 /build/buildd/eglibc-2.15/math/../sysdeps/i386/fpu/e_exp.S:__ieee754_exp [/lib/i386-linux-gnu/libm-2.15.so]
181,517,469 ???:operator new(unsigned int) [/usr/lib/i386-linux-gnu/libstdc++.so.6.0.16]
171,582,030 /usr/include/c++/4.6/bits/stl_iterator_base_types.h:std::_Iter_base<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, true>::_S_base(__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >) [a.out]
171,581,742 /build/buildd/eglibc-2.15/math/../sysdeps/ieee754/dbl-64/w_exp.c:exp [/lib/i386-linux-gnu/libm-2.15.so]
152,853,344 ???:__gnu_cxx::new_allocator<double>::allocate(unsigned int, void const*) [a.out]
152,852,752 ???:std::_Vector_base<double, std::allocator<double> >::_Vector_impl::_Vector_impl(std::allocator<double> const&) [a.out]
152,517,360 /usr/include/c++/4.6/bits/stl_algobase.h:std::_Niter_base<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > > >::iterator_type std::__niter_base<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > > >(__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >) [a.out]
152,517,360 /usr/include/c++/4.6/bits/stl_iterator_base_types.h:std::_Iter_base<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, false>::_S_base(__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >) [a.out]
152,517,360 /usr/include/c++/4.6/bits/stl_iterator.h:__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >::__normal_iterator(double const* const&) [a.out]
133,746,571 ???:std::_Vector_base<double, std::allocator<double> >::_M_deallocate(double*, unsigned int) [a.out]
133,452,690 ???:std::vector<double, std::allocator<double> >::end() const [a.out]
133,452,690 ???:std::vector<double, std::allocator<double> >::begin() const [a.out]
131,134,604 sgd.cpp:sign(double) [a.out]
123,920,353 /usr/include/c++/4.6/bits/stl_algobase.h:double* std::__copy_move_a<false, double const*, double*>(double const*, double const*, double*) [a.out]
121,192,848 ???:std::vector<int, std::allocator<int> >::operator[](unsigned int) [a.out]
114,649,360 /build/buildd/eglibc-2.15/string/../sysdeps/i386/i686/multiarch/strcat.S:0x0012ac73 [/lib/i386-linux-gnu/libc-2.15.so]
114,642,456 ???:operator delete(void*) [/usr/lib/i386-linux-gnu/libstdc++.so.6.0.16]
114,388,018 /usr/include/c++/4.6/bits/stl_uninitialized.h:double* std::__uninitialized_copy<true>::__uninit_copy<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, double*>(__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, double*) [a.out]
114,388,018 /usr/include/c++/4.6/bits/stl_uninitialized.h:double* std::uninitialized_copy<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, double*>(__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, double*) [a.out]
114,388,018 /usr/include/c++/4.6/bits/stl_uninitialized.h:double* std::__uninitialized_copy_a<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, double*, double>(__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, double*, std::allocator<double>&) [a.out]
105,086,674 /usr/include/c++/4.6/bits/stl_vector.h:std::_Vector_base<double, std::allocator<double> >::_M_allocate(unsigned int) [a.out]
95,533,505 ???:std::_Vector_base<double, std::allocator<double> >::_M_get_Tp_allocator() [a.out]
95,533,300 /usr/include/c++/4.6/bits/stl_construct.h:void std::_Destroy<double*>(double*, double*) [a.out]
95,533,300 /usr/include/c++/4.6/bits/stl_construct.h:void std::_Destroy<double*, double>(double*, double*, std::allocator<double>&) [a.out]
95,532,970 /usr/include/c++/4.6/bits/allocator.h:std::allocator<double>::allocator(std::allocator<double> const&) [a.out]
95,323,350 /usr/include/c++/4.6/bits/stl_iterator.h:__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >::base() const [a.out]
76,594,040 /usr/include/c++/4.6/bits/allocator.h:std::allocator<double>::~allocator() [a.out]
76,428,152 /usr/include/c++/4.6/bits/stl_algobase.h:std::_Niter_base<double*>::iterator_type std::__niter_base<double*>(double*) [a.out]
76,426,584 /usr/include/c++/4.6/ext/new_allocator.h:__gnu_cxx::new_allocator<double>::deallocate(double*, unsigned int) [a.out]
76,426,344 ???:std::_Vector_base<double, std::allocator<double> >::_Vector_impl::~_Vector_impl() [a.out]
75,798,592 /usr/include/c++/4.6/bits/stl_algobase.h:__gnu_cxx::__enable_if<std::__is_scalar<double>::__value, double*>::__type std::__fill_n_a<double*, unsigned int, double>(double*, unsigned int, double const&) [a.out]
47,768,335 /usr/include/c++/4.6/bits/stl_iterator_base_types.h:std::_Iter_base<double*, false>::_S_base(double*) [a.out]
47,767,040 ???:__gnu_cxx::new_allocator<double>::max_size() const [a.out]
47,662,045 ???:std::_Vector_base<double, std::allocator<double> >::_M_get_Tp_allocator() const [a.out]
38,297,020 /usr/include/c++/4.6/ext/new_allocator.h:__gnu_cxx::new_allocator<double>::~new_allocator() [a.out]
This has more information than the previous output, but there are still two issues: One, output on unoptimized code doesn't help me make the optimized code faster. Two, most of the time (~50%) is taken up by libc functions, which I am not directly using in my code. How do I know which parts of the code map to these calls?
Related
I'm running a piece of code which allocates memory using std::shared_ptr<>. However, on shutting down the process, GCC ASAN complaints of things like Indirect leak of 91723904 byte(s) in 383 object(s) and there are no direct leaks.
Error -
Indirect leak of 91723904 byte(s) in 383 object(s) allocated from:
#0 0x7f93658497c0 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xef7c0)
#1 0xc0275b in __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<Horizon::Zone::Game::Map, std::allocator<Horizon::Zone::Game::Map>, (__gnu_cxx::_Lock_policy)2> >::allocate(unsigned long, void const*) /usr/include/c++/8/ext/new_allocator.h:111
#2 0xc01eaf in std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<Horizon::Zone::Game::Map, std::allocator<Horizon::Zone::Game::Map>, (__gnu_cxx::_Lock_policy)2> > >::allocate(std::allocator<std::_Sp_counted_ptr_inplace<Horizon::Zone::Game::Map, std::allocator<Horizon::Zone::Game::Map>, (__gnu_cxx::_Lock_policy)2> >&, unsigned long) /usr/include/c++/8/bits/alloc_traits.h:436
#3 0xc01087 in std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<Horizon::Zone::Game::Map, std::allocator<Horizon::Zone::Game::Map>, (__gnu_cxx::_Lock_policy)2> > > std::__allocate_guarded<std::allocator<std::_Sp_counted_ptr_inplace<Horizon::Zone::Game::Map, std::allocator<Horizon::Zone::Game::Map>, (__gnu_cxx::_Lock_policy)2> > >(std::allocator<std::_Sp_counted_ptr_inplace<Horizon::Zone::Game::Map, std::allocator<Horizon::Zone::Game::Map>, (__gnu_cxx::_Lock_policy)2> >&) /usr/include/c++/8/bits/allocated_ptr.h:97
#4 0xc000d3 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<Horizon::Zone::Game::Map, std::allocator<Horizon::Zone::Game::Map>, std::shared_ptr<Horizon::Zone::Game::MapThreadContainer>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned short, unsigned short, std::vector<unsigned char, std::allocator<unsigned char> > const&>(std::_Sp_make_shared_tag, Horizon::Zone::Game::Map*, std::allocator<Horizon::Zone::Game::Map> const&, std::shared_ptr<Horizon::Zone::Game::MapThreadContainer>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, unsigned short&&, unsigned short&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) /usr/include/c++/8/bits/shared_ptr_base.h:660
#5 0xbfeda1 in std::__shared_ptr<Horizon::Zone::Game::Map, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<Horizon::Zone::Game::Map>, std::shared_ptr<Horizon::Zone::Game::MapThreadContainer>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned short, unsigned short, std::vector<unsigned char, std::allocator<unsigned char> > const&>(std::_Sp_make_shared_tag, std::allocator<Horizon::Zone::Game::Map> const&, std::shared_ptr<Horizon::Zone::Game::MapThreadContainer>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, unsigned short&&, unsigned short&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) /usr/include/c++/8/bits/shared_ptr_base.h:1328
#6 0xbfd2be in std::shared_ptr<Horizon::Zone::Game::Map>::shared_ptr<std::allocator<Horizon::Zone::Game::Map>, std::shared_ptr<Horizon::Zone::Game::MapThreadContainer>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned short, unsigned short, std::vector<unsigned char, std::allocator<unsigned char> > const&>(std::_Sp_make_shared_tag, std::allocator<Horizon::Zone::Game::Map> const&, std::shared_ptr<Horizon::Zone::Game::MapThreadContainer>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, unsigned short&&, unsigned short&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (/home/travis/build/horizonxyz/horizon/build/bin/Debug/zone+0xbfd2be)
#7 0xbfb57b in std::shared_ptr<Horizon::Zone::Game::Map> std::allocate_shared<Horizon::Zone::Game::Map, std::allocator<Horizon::Zone::Game::Map>, std::shared_ptr<Horizon::Zone::Game::MapThreadContainer>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned short, unsigned short, std::vector<unsigned char, std::allocator<unsigned char> > const&>(std::allocator<Horizon::Zone::Game::Map> const&, std::shared_ptr<Horizon::Zone::Game::MapThreadContainer>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, unsigned short&&, unsigned short&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (/home/travis/build/horizonxyz/horizon/build/bin/Debug/zone+0xbfb57b)
#8 0xbfa076 in std::shared_ptr<Horizon::Zone::Game::Map> std::make_shared<Horizon::Zone::Game::Map, std::shared_ptr<Horizon::Zone::Game::MapThreadContainer>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned short, unsigned short, std::vector<unsigned char, std::allocator<unsigned char> > const&>(std::shared_ptr<Horizon::Zone::Game::MapThreadContainer>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, unsigned short&&, unsigned short&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) /usr/include/c++/8/bits/shared_ptr.h:723
#9 0xbf4371 in Horizon::Zone::Game::MapManager::LoadMapCache() /home/travis/build/horizonxyz/horizon/src/Server/Zone/Game/Map/MapManager.cpp:107
#10 0xbf06f6 in Horizon::Zone::Game::MapManager::initialize() /home/travis/build/horizonxyz/horizon/src/Server/Zone/Game/Map/MapManager.cpp:49
#11 0x42e82b in Horizon::Zone::ZoneMain::initialize_core() /home/travis/build/horizonxyz/horizon/src/Server/Zone/Zone.cpp:145
#12 0x42f656 in main /home/travis/build/horizonxyz/horizon/src/Server/Zone/Zone.cpp:223
#13 0x7f936372982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
As seen above this is allocated using shared_ptr which doesn't require objects to be explicitly freed. But then again, there would need to be an internal call from the STL to delete the object which I think isn't happening as the process ends? Or that GCC isn't able to capture it.
I used valgrind to check my project written in C++. It told me there was a definitely memory leak in a line with vector assignment.
What valgrind told is
==37533== 514,604 bytes in 124,984 blocks are definitely lost in loss record 369 of 433
==37533== at 0x4C2E0EF: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==37533== by 0x42BB89: __gnu_cxx::new_allocator<int>::allocate(unsigned long, void const*) (new_allocator.h:104)
==37533== by 0x42832B: std::allocator_traits<std::allocator<int> >::allocate(std::allocator<int>&, unsigned long) (alloc_traits.h:491)
==37533== by 0x424565: std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) (stl_vector.h:170)
==37533== by 0x48B73F: int* std::vector<int, std::allocator<int> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > > >(unsigned long, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >) (stl_vector.h:1224)
==37533== by 0x48B2BD: std::vector<int, std::allocator<int> >::operator=(std::vector<int, std::allocator<int> > const&) (vector.tcc:195)
==37533== by 0x603A49: Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::buildTrie(std::vector<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TermCode, std::allocator<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TermCode> > const&, std::vector<std::vector<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TrieNode, std::allocator<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TrieNode> >*, std::allocator<std::vector<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TrieNode, std::allocator<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TrieNode> >*> >&) (Datrie.cpp:467)
==37533== by 0x602758: Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::build(std::vector<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TermCode, std::allocator<Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::TermCode> > const&) (Datrie.cpp:178)
==37533== by 0x602103: Loader::Trie_Loader::InnerLoad(BaseDict_Collection&) (Trie_Loader.cpp:116)
==37533== by 0x473BA6: deduce::DeduceClassify::Get_Dat_EntityRec() (DeduceClassify.cpp:500)
==37533== by 0x470D7A: deduce::DeduceClassify::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<Qp_Dict> const&, Analyse::Analysis*, MINISEARCH::Index*, Normalization::Normalizer*) (DeduceClassify.cpp:53)
==37533== by 0x40E784: queryparser::CQueryParserImpl::Init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (QueryParserImpl.cpp:120)
The code snippet is
if ((int) (*itVecIter)->code.size() == nLevel + 1)
{
treeNode.bEnd = true;
// NOTE: the memory leak happens at following
treeNode.weight = (*itVecIter)->weight;
itVecIter = vecIter.erase(itVecIter);
}
And the runtime callstack via gdb is
(gdb) bt
#0 Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::buildTrie (this=0x7fffffffd8f8, vkey_code=std::vector of length 125163, capacity 131072 = {...},
trie=std::vector of length 31422, capacity 31422 = {...}) at ./tools/Dict/Datrie.cpp:467
#1 0x0000000000602759 in Dict::DATrieT<unsigned short, std::vector<unsigned short, std::allocator<unsigned short> >, std::vector<int, std::allocator<int> > >::build (this=0x7fffffffd8f8, vkey_code=std::vector of length 125163, capacity 131072 = {...})
at ./tools/Dict/Datrie.cpp:178
#2 0x0000000000602104 in Loader::Trie_Loader::InnerLoad (this=0x7fffffffd8e0, dictPtrs=...) at loader//Trie_Loader.cpp:116
#3 0x0000000000473ba7 in deduce::DeduceClassify::Get_Dat_EntityRec (this=0xa2fb28) at Classification/Deduce//DeduceClassify.cpp:500
#4 0x0000000000470d7b in deduce::DeduceClassify::init (this=0xa2fb28, base_Path="..//data/",
qp_dict=std::shared_ptr (count 2, weak 0) 0x1bc14600, tokenizer=0xa2f998, indexer=0xf6efb0, pNormer=0xa2f898)
at Classification/Deduce//DeduceClassify.cpp:53
#5 0x000000000040e785 in queryparser::CQueryParserImpl::Init (this=0xa2f4a0, dataPath="../") at QueryParserImpl.cpp:120
#6 0x000000000043ca92 in main_process::test_main (argc=2, argv=0x7fffffffdd20) at MainProcess.cpp:284
#7 0x00000000006301ae in main (argc=3, argv=0x7fffffffde38) at main.cpp:57
I used template class in Datrie.cpp. And I am not sure why vector assignment with
treeNode.weight = (*itVecIter)->weight;
has memory leaks.
Thanks.
It seems that the assignment operator of treeNode.weight is called, it allocates some memory to use but never de-allocated.
==37533== by 0x48B73F: int* std::vector<int, std::allocator<int> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > > >(unsigned long, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >) (stl_vector.h:1224)
==37533== by 0x48B2BD: std::vector<int, std::allocator<int> >::operator=(std::vector<int, std::allocator<int> > const&) (vector.tcc:195)
As #n.m.'s suggestion, I've checked the code whole around. And I think I found the issue. Simplified the class constructor/destructor and manipulation in following:
#include <iostream>
#include <vector>
#include <string.h>
using namespace std;
struct leak_struct {
vector<string> array;
int set;
};
int main(){
leak_struct ls;
ls.array.push_back("hello");
ls.array.push_back("world");
memset(&ls, 0, sizeof(ls));
ls.array.push_back("code");
ls.array.push_back("review");
memset(&ls, 0, sizeof(ls));
return 0;
}
The reason it leaks is at the line where memset is called, this call will nullify the vector pointer in struct and makes the new allocated buffer not freed.
I have this method: (it's supposed to create a inverted list from all the files in the given directory)
class Index{
public:
Index();
void create();
void writeInvertedIndex();
private:
bool isWhiteSpace(const char ch);
std::map<std::string, std::set<int>> invertedIndex;
};
void Index::create(){
std::string datasetPath = "/home/skluzada/Downloads/BI-VWM/Project/dataset/";
std::string filePath, word, text;
std::ifstream infile;
int fileIndex = 0;
std::size_t textLen, i;
DIR * dir;
struct dirent * ent;
if ((dir = opendir (datasetPath.c_str())) != NULL){
while((ent = readdir(dir)) != NULL){
filePath = datasetPath + ent->d_name;
std::cout << filePath << std::endl;
std::ifstream inFile(filePath, std::ios::in);
std::stringstream buffer;
buffer << inFile.rdbuf();
std::string text = buffer.str();
inFile.close();
textLen = text.size();
i = 19;
while (i < textLen){
word = "";
while(isWhiteSpace(text[i])){
i++;
}
while(!isWhiteSpace(text[i])){
word = word + text[i];
i++;
}
invertedIndex[word].insert(fileIndex);
}
fileIndex++;
}
}
}
When I run the program on a small collection of files (around 50Kb of text) it works fine, but when I run it on the real collection (500 files around 1.5Mb in total) it segfaults after processing around 50 files. The output inverted index is about 900Kb.
I suppose that loading the whole file into a string is not an ideal solution (the files are 1-5Kb each) but I'm using a similar approach in other parts of the project and it works fine even on the whole collection.
Could you suggest where is the problem? Or maybe give me some advice on optimization?
The weirdest thing about this is that when I run it with Valgrind it processes the whole collection without segfaulting. This is the Valgrind output:
==9952== Syscall param writev(vector[...]) points to uninitialised byte(s)
==9952== at 0x57F6610: __writev_nocancel (in /usr/lib64/libc-2.25.so)
==9952== by 0x4EEC4B1: std::__basic_file<char>::xsputn_2(char const*, long, char const*, long) (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x4F29BC1: std::basic_filebuf<char, std::char_traits<char> >::xsputn(char const*, long) (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x4F4E063: std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long) (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x401BFC: Index::writeInvertedIndex() (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x4021D0: main (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== Address 0x6f929f4 is 84 bytes inside a block of size 2,273 alloc'd
==9952== at 0x4C2E1CA: operator new(unsigned long) (vg_replace_malloc.c:334)
==9952== by 0x4F62144: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x4F6219E: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x404A5F: std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > >::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, 0ul>(std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&, std::tuple<>&, std::_Index_tuple<0ul>, std::_Index_tuple<>) (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x404782: std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > >::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(std::piecewise_construct_t, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<>) (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x40458C: void __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > > >::construct<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > >, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > >*, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x404247: void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > > > >::construct<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > >, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > > >&, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > >*, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x403C6C: void std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > > >::_M_construct_node<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > >*, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x403069: std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x402C4D: std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > > std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > >, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x4028A4: std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<int, std::less<int>, std::allocator<int> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<int, std::less<int>, std::allocator<int> > > > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x40203A: Index::create() (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952==
==9952== Syscall param writev(vector[...]) points to uninitialised byte(s)
==9952== at 0x57F6610: __writev_nocancel (in /usr/lib64/libc-2.25.so)
==9952== by 0x4EEC4B1: std::__basic_file<char>::xsputn_2(char const*, long, char const*, long) (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x4F29BC1: std::basic_filebuf<char, std::char_traits<char> >::xsputn(char const*, long) (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x4F42581: std::ostreambuf_iterator<char, std::char_traits<char> > std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>(std::ostreambuf_iterator<char, std::char_traits<char> >, std::ios_base&, char, long) const (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x4F4E564: std::ostream& std::ostream::_M_insert<long>(long) (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x401C82: Index::writeInvertedIndex() (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x4021D0: main (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== Address 0x5d0f861 is 721 bytes inside a block of size 8,192 alloc'd
==9952== at 0x4C2E8B7: operator new[](unsigned long) (vg_replace_malloc.c:423)
==9952== by 0x4F2AA87: std::basic_filebuf<char, std::char_traits<char> >::_M_allocate_internal_buffer() (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x4F2EC71: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x4F2ED92: std::basic_ofstream<char, std::char_traits<char> >::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode) (in /usr/lib64/libstdc++.so.6.0.24)
==9952== by 0x401B75: Index::writeInvertedIndex() (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952== by 0x4021D0: main (in /home/skluzada/Downloads/BI-VWM/Project/index)
==9952==
==9952==
==9952== HEAP SUMMARY:
==9952== in use at exit: 32,816 bytes in 1 blocks
==9952== total heap usage: 820,861 allocs, 820,860 frees, 16,188,505,659 bytes allocated
==9952==
==9952== LEAK SUMMARY:
==9952== definitely lost: 32,816 bytes in 1 blocks
==9952== indirectly lost: 0 bytes in 0 blocks
==9952== possibly lost: 0 bytes in 0 blocks
==9952== still reachable: 0 bytes in 0 blocks
==9952== suppressed: 0 bytes in 0 blocks
==9952== Rerun with --leak-check=full to see details of leaked memory
==9952==
==9952== For counts of detected and suppressed errors, rerun with: -v
==9952== Use --track-origins=yes to see where uninitialised values come from
==9952== ERROR SUMMARY: 681764 errors from 10 contexts (suppressed: 0 from 0)
The solution for the asked question is the comment from rafix07:
Why don't you check i < textLen in while(isWhiteSpace(text[i])){i++;} and in next while loop, you probably read data out of range. Change to while (i < textLen && isWhiteSpace[text[i]]){i++;} and do the same in second loop.
This question already has answers here:
Is it smart to replace boost::thread and boost::mutex with c++11 equivalents?
(7 answers)
Closed 9 years ago.
have a code which used to run fine with boost thread under ubuntu. it's basic read only data sharing multithreading. I try to use C++11 instead of boost, very basic transition. the code compiles but have subtle bugs. crashes randomly with C++11 std thread. tried to use valgrind drd, but hard to read the debug info. Any thoughts?
==19608== Conflicting load by thread 3 at 0x00643be8 size 8
==19608== at 0x41CEBA: std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul>::operator()() (random.tcc:601)
==19608== by 0x41CD6C: double std::generate_canonical<double, 53ul, std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul> >(std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul>&) (random.tcc:3475)
==19608== by 0x41CB06: std::__detail::_Adaptor<std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul>, double>::operator()() (random.h:190)
==19608== by 0x41C877: double std::normal_distribution<double>::operator()<std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul> >(std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul>&, std::normal_distribution<double>::param_type const&) (random.tcc:1950)
==19608== by 0x41C688: double std::normal_distribution<double>::operator()<std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul> >(std::subtract_with_carry_engine<unsigned int, 24ul, 10ul, 24ul>&) (random.h:2196)
==19608== by 0x41C379: nrand(double, double) (NRand.cpp:8)
==19608== by 0x416F78: ClassDef::set_x() (LoanDef.h:407)
==19608== by 0x4168CA: Sim(std::vector<ClassDef, std::allocator<ClassDef> >&, Assumption&, std::unordered_map<std::string, std::vector<double, std::allocator<double> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<double, std::allocator<double> > > > >&, unsigned int, NextStepCalcML&, std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >&, unsigned int, unsigned int) (NewSim.cpp:105)
==19608== by 0x411A09: _ZNSt12_Bind_simpleIFPFvRSt6vectorI7LoanDefSaIS1_EER10AssumptionRSt13unordered_mapISsS0_IdSaIdEESt4hashISsESt8equal_toISsESaISt4pairIKSsS9_EEEjR14NextStepCalcMLRS0_IS0_IS9_SaIS9_EESaISN_EEjjESt17reference_wrapperIS3_EST_IS5_EST_ISI_EjST_ISK_EST_ISP_EjjEE9_M_invokeIILm0ELm1ELm2ELm3ELm4ELm5ELm6ELm7EEEEvSt12_Index_tupleIIXspT_EEE (functional:1732)
==19608== by 0x4116AE: std::_Bind_simple<void (*()(std::reference_wrapper<std::vector<LoanDef, std::allocator<LoanDef> > >, std::reference_wrapper<Assumption>, std::reference_wrapper<std::unordered_map<std::string, std::vector<double, std::allocator<double> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<double, std::allocator<double> > > > > >, unsigned int, std::reference_wrapper<NextStepCalcML>, std::reference_wrapper<std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > > >, unsigned int, unsigned int))(std::vector<LoanDef, std::allocator<ClassDef> >&, Assumption&, std::unordered_map<std::string, std::vector<double, std::allocator<double> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<double, std::allocator<double> > > > >&, unsigned int, NextStep&, std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >&, unsigned int, unsigned int)>::operator()() (functional:1720)
==19608== by 0x411647: std::thread::_Impl<std::_Bind_simple<void (*()(std::reference_wrapper<std::vector<ClassDef, std::allocator<ClassDef> > >, std::reference_wrapper<Assumption>, std::reference_wrapper<std::unordered_map<std::string, std::vector<double, std::allocator<double> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<double, std::allocator<double> > > > > >, unsigned int, std::reference_wrapper<NextStep>, std::reference_wrapper<std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > > >, unsigned int, unsigned int))(std::vector<LoanDef, std::allocator<LoanDef> >&, Assumption&, std::unordered_map<std::string, std::vector<double, std::allocator<double> >, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::vector<double, std::allocator<double> > > > >&, unsigned int, NextStep&, std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >&, unsigned int, unsigned int)> >::_M_run() (thread:115)
thank you. btw, I tried to use this random head file I wrote, not sure if it's safe under multithread environment. it used to work fine with boost.
myrand.hpp
#ifndef NRAND_H
#define NRAND_H
#include <random>
double nrand(double mean = 0., double sd = 1.);
double urand(double a=0., double b=0.);
#endif
and myrand.cpp
#include "NRand.h"
using namespace std;
double nrand(double mean, double sd) {
static random_device rd;
static subtract_with_carry_engine<unsigned,24,10,24> e(rd());
normal_distribution<> dist(mean, sd);
return dist(e);
}
double urand(double a, double b) {
static random_device rd;
static subtract_with_carry_engine<unsigned,24,10,24> e(rd());
uniform_real_distribution<> dist(a, b);
return dist(e);
}
many thanks.
IMHO, your code is not threadsafe and hence should not work correctly under C++11. I think the problem is that the static variables rd and e are global variables but not protected (by mutex), so concurrent calls will race.
Presumably, you can make this code threadsafe by making those variables thread_local, but I have no experience.
I am trying to call the following C++ function from Fortran 90.
//Filename ctest.cpp
#include<iostream>
#include<vector>
extern "C"
{
extern struct{
std::vector< std::vector<double> > a;
std::vector< std::vector<double> > b;
std::vector< std::vector<double> > c;
}abc_;
}
int myfunc_(int y,int z)
{
std::vector< std::vector<double> > u(y,std::vector<double>(z,2.0));
std::vector< std::vector<double> > v(y,std::vector<double>(z,4.0));
std::vector< std::vector<double> > w(y,std::vector<double>(z,6.0));
abc_.a = u;
abc_.b = v;
abc_.c = w;
return(1);
}
The corresponding fortran code is
! File fortest.f90
! Fortran test code to interface with C++ code
! return struct from C++
program fortest
implicit none
common/abc/ a,b,c
double precision,dimension(10,10) :: a
double precision,dimension(10,10) :: b
double precision,dimension(10,10) :: c
integer y,z
y = 10
z = 10
call myfunc(y,z)
write(*,*) a,b,c
stop
end
Both codes compile without any problems individually. However, when I compile the 2 together to interface them, with
gfortran -o test fortest.o ctest.o
I get a really huge error message. I suspect the problem is with Fortran not recognizing C++ 2d std::vector. But I am enclosing the full message here in case anyone wants to have a look.
user#userpc$ gfortran -o test fortest.o ctest.o
fortest.o: In function `MAIN__':
fortest.f90:(.text+0x2d): undefined reference to `myfunc_'
ctest.o: In function `__static_initialization_and_destruction_0(int, int)':
ctest.cpp:(.text+0x379): undefined reference to `std::ios_base::Init::Init()'
ctest.cpp:(.text+0x37e): undefined reference to `std::ios_base::Init::~Init()'
ctest.o: In function `std::vector<double, std::allocator<double> >* std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >(unsigned long, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >)':
ctest.cpp:(.text._ZNSt6vectorIS_IdSaIdEESaIS1_EE20_M_allocate_and_copyIN9__gnu_cxx17__normal_iteratorIPKS1_S3_EEEEPS1_mT_SB_[std::vector<double, std::allocator<double> >* std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >(unsigned long, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >)]+0x62): undefined reference to `__cxa_end_catch'
ctest.cpp:(.text._ZNSt6vectorIS_IdSaIdEESaIS1_EE20_M_allocate_and_copyIN9__gnu_cxx17__normal_iteratorIPKS1_S3_EEEEPS1_mT_SB_[std::vector<double, std::allocator<double> >* std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >(unsigned long, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >)]+0x75): undefined reference to `__cxa_begin_catch'
ctest.cpp:(.text._ZNSt6vectorIS_IdSaIdEESaIS1_EE20_M_allocate_and_copyIN9__gnu_cxx17__normal_iteratorIPKS1_S3_EEEEPS1_mT_SB_[std::vector<double, std::allocator<double> >* std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >(unsigned long, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >)]+0x91): undefined reference to `__cxa_rethrow'
ctest.o: In function `__gnu_cxx::new_allocator<std::vector<double, std::allocator<double> > >::deallocate(std::vector<double, std::allocator<double> >*, unsigned long)':
ctest.cpp:(.text._ZN9__gnu_cxx13new_allocatorISt6vectorIdSaIdEEE10deallocateEPS3_m[__gnu_cxx::new_allocator<std::vector<double, std::allocator<double> > >::deallocate(std::vector<double, std::allocator<double> >*, unsigned long)]+0x1c): undefined reference to `operator delete(void*)'
ctest.o: In function `__gnu_cxx::new_allocator<double>::allocate(unsigned long, void const*)':
ctest.cpp:(.text._ZN9__gnu_cxx13new_allocatorIdE8allocateEmPKv[__gnu_cxx::new_allocator<double>::allocate(unsigned long, void const*)]+0x2c): undefined reference to `std::__throw_bad_alloc()'
ctest.cpp:(.text._ZN9__gnu_cxx13new_allocatorIdE8allocateEmPKv[__gnu_cxx::new_allocator<double>::allocate(unsigned long, void const*)]+0x3c): undefined reference to `operator new(unsigned long)'
ctest.o: In function `__gnu_cxx::new_allocator<double>::deallocate(double*, unsigned long)':
ctest.cpp:(.text._ZN9__gnu_cxx13new_allocatorIdE10deallocateEPdm[__gnu_cxx::new_allocator<double>::deallocate(double*, unsigned long)]+0x1c): undefined reference to `operator delete(void*)'
ctest.o: In function `__gnu_cxx::new_allocator<std::vector<double, std::allocator<double> > >::allocate(unsigned long, void const*)':
ctest.cpp:(.text._ZN9__gnu_cxx13new_allocatorISt6vectorIdSaIdEEE8allocateEmPKv[__gnu_cxx::new_allocator<std::vector<double, std::allocator<double> > >::allocate(unsigned long, void const*)]+0x2c): undefined reference to `std::__throw_bad_alloc()'
ctest.cpp:(.text._ZN9__gnu_cxx13new_allocatorISt6vectorIdSaIdEEE8allocateEmPKv[__gnu_cxx::new_allocator<std::vector<double, std::allocator<double> > >::allocate(unsigned long, void const*)]+0x45): undefined reference to `operator new(unsigned long)'
ctest.o: In function `std::vector<double, std::allocator<double> >* std::__uninitialized_copy<false>::__uninit_copy<std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*>(std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*)':
ctest.cpp:(.text._ZNSt20__uninitialized_copyILb0EE13__uninit_copyIPSt6vectorIdSaIdEES5_EET0_T_S7_S6_[std::vector<double, std::allocator<double> >* std::__uninitialized_copy<false>::__uninit_copy<std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*>(std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*)]+0x62): undefined reference to `__cxa_end_catch'
ctest.cpp:(.text._ZNSt20__uninitialized_copyILb0EE13__uninit_copyIPSt6vectorIdSaIdEES5_EET0_T_S7_S6_[std::vector<double, std::allocator<double> >* std::__uninitialized_copy<false>::__uninit_copy<std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*>(std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*)]+0x75): undefined reference to `__cxa_begin_catch'
ctest.cpp:(.text._ZNSt20__uninitialized_copyILb0EE13__uninit_copyIPSt6vectorIdSaIdEES5_EET0_T_S7_S6_[std::vector<double, std::allocator<double> >* std::__uninitialized_copy<false>::__uninit_copy<std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*>(std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*, std::vector<double, std::allocator<double> >*)]+0x8d): undefined reference to `__cxa_rethrow'
ctest.o: In function `void std::__uninitialized_fill_n<false>::__uninit_fill_n<std::vector<double, std::allocator<double> >*, unsigned long, std::vector<double, std::allocator<double> > >(std::vector<double, std::allocator<double> >*, unsigned long, std::vector<double, std::allocator<double> > const&)':
ctest.cpp:(.text._ZNSt22__uninitialized_fill_nILb0EE15__uninit_fill_nIPSt6vectorIdSaIdEEmS4_EEvT_T0_RKT1_[void std::__uninitialized_fill_n<false>::__uninit_fill_n<std::vector<double, std::allocator<double> >*, unsigned long, std::vector<double, std::allocator<double> > >(std::vector<double, std::allocator<double> >*, unsigned long, std::vector<double, std::allocator<double> > const&)]+0x56): undefined reference to `__cxa_end_catch'
ctest.cpp:(.text._ZNSt22__uninitialized_fill_nILb0EE15__uninit_fill_nIPSt6vectorIdSaIdEEmS4_EEvT_T0_RKT1_[void std::__uninitialized_fill_n<false>::__uninit_fill_n<std::vector<double, std::allocator<double> >*, unsigned long, std::vector<double, std::allocator<double> > >(std::vector<double, std::allocator<double> >*, unsigned long, std::vector<double, std::allocator<double> > const&)]+0x69): undefined reference to `__cxa_begin_catch'
ctest.cpp:(.text._ZNSt22__uninitialized_fill_nILb0EE15__uninit_fill_nIPSt6vectorIdSaIdEEmS4_EEvT_T0_RKT1_[void std::__uninitialized_fill_n<false>::__uninit_fill_n<std::vector<double, std::allocator<double> >*, unsigned long, std::vector<double, std::allocator<double> > >(std::vector<double, std::allocator<double> >*, unsigned long, std::vector<double, std::allocator<double> > const&)]+0x81): undefined reference to `__cxa_rethrow'
ctest.o: In function `std::vector<double, std::allocator<double> >* std::__uninitialized_copy<false>::__uninit_copy<__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, std::vector<double, std::allocator<double> >*>(__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, std::vector<double, std::allocator<double> >*)':
ctest.cpp:(.text._ZNSt20__uninitialized_copyILb0EE13__uninit_copyIN9__gnu_cxx17__normal_iteratorIPKSt6vectorIdSaIdEES4_IS6_SaIS6_EEEEPS6_EET0_T_SE_SD_[std::vector<double, std::allocator<double> >* std::__uninitialized_copy<false>::__uninit_copy<__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, std::vector<double, std::allocator<double> >*>(__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, std::vector<double, std::allocator<double> >*)]+0x7c): undefined reference to `__cxa_end_catch'
ctest.cpp:(.text._ZNSt20__uninitialized_copyILb0EE13__uninit_copyIN9__gnu_cxx17__normal_iteratorIPKSt6vectorIdSaIdEES4_IS6_SaIS6_EEEEPS6_EET0_T_SE_SD_[std::vector<double, std::allocator<double> >* std::__uninitialized_copy<false>::__uninit_copy<__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, std::vector<double, std::allocator<double> >*>(__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, std::vector<double, std::allocator<double> >*)]+0x8f): undefined reference to `__cxa_begin_catch'
ctest.cpp:(.text._ZNSt20__uninitialized_copyILb0EE13__uninit_copyIN9__gnu_cxx17__normal_iteratorIPKSt6vectorIdSaIdEES4_IS6_SaIS6_EEEEPS6_EET0_T_SE_SD_[std::vector<double, std::allocator<double> >* std::__uninitialized_copy<false>::__uninit_copy<__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, std::vector<double, std::allocator<double> >*>(__gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, __gnu_cxx::__normal_iterator<std::vector<double, std::allocator<double> > const*, std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > >, std::vector<double, std::allocator<double> >*)]+0xa7): undefined reference to `__cxa_rethrow'
ctest.o: In function `double* std::vector<double, std::allocator<double> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > > >(unsigned long, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >)':
ctest.cpp:(.text._ZNSt6vectorIdSaIdEE20_M_allocate_and_copyIN9__gnu_cxx17__normal_iteratorIPKdS1_EEEEPdmT_S9_[double* std::vector<double, std::allocator<double> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > > >(unsigned long, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >)]+0x62): undefined reference to `__cxa_end_catch'
ctest.cpp:(.text._ZNSt6vectorIdSaIdEE20_M_allocate_and_copyIN9__gnu_cxx17__normal_iteratorIPKdS1_EEEEPdmT_S9_[double* std::vector<double, std::allocator<double> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > > >(unsigned long, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >)]+0x75): undefined reference to `__cxa_begin_catch'
ctest.cpp:(.text._ZNSt6vectorIdSaIdEE20_M_allocate_and_copyIN9__gnu_cxx17__normal_iteratorIPKdS1_EEEEPdmT_S9_[double* std::vector<double, std::allocator<double> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > > >(unsigned long, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >, __gnu_cxx::__normal_iterator<double const*, std::vector<double, std::allocator<double> > >)]+0x91): undefined reference to `__cxa_rethrow'
ctest.o:(.eh_frame+0x6b): undefined reference to `__gxx_personality_v0'
collect2: ld returned 1 exit status
Any idea whats wrong.... in general terms how do I call multidimensional std::vectors from C++ in fortran? This will be of great use to people doing computational sciences.
std::vector is an object that can be expected to have various data members such as capacity, pointer-to-start-of-data, pointer-to-end etc.. The exact layout is not specified by the Standard - implementations may vary. The data is not stored directly in the vector - normally the pointer-to-start-of-data addresses heap-allocated memory. Therefore, you can't just point a fortran array at the vector object, as it will expect double data to be contiguously in memory from that address. You can't even use the .data() member to get the address where the pointer points, as in your case that's another vector and has the same problem. Put another way, your data just isn't contiguous in memory so won't correspond to the memory layout for a 2-dimensional array.
You could try copying the data into a packed two-dimensional array: if you know the dimensions will always be 10x10, then you can use a stack allocated area such as double data[10][10], otherwise you can double* p = new double[100] and put the logical value for [x][y] into element [x*10+y]. I'm guessing that's what your fortran implementation will expect but don't know for sure. It could be that it will expect contiguous data but packs it using [x+10*y], or it's even possible that it might introduce some other padding. It's worth checking with your fortran documentation. If all else fails (or you just think it's fun) you should be able to work it out by putting data into the fortran array and looking at the memory in a debugger to work out the layout.