Related
I am writing a multi-threaded XML parser using Xercesc and I have followed the guide in https://xerces.apache.org/xerces-c/faq-parse-3.html#faq-6 by declaring the XMLPlatformUtils::Initialize() and XMLPlatformUtils::Terminate(); in the main thread while creating new parser and handler that can parse different XML files concurrently. The threads are created this way
std::vector<std::thread> thread_vector;
for (std::string i: input_folder){
SAXParser* parser = new SAXParser();
parser->setValidationScheme(SAXParser::ValSchemes::Val_Never);
parser->setDoSchema(false);
parser->setValidationSchemaFullChecking(false);
parser->cacheGrammarFromParse(false);
parser->setDoNamespaces(true); // optional
SAXHandler* handler= new SAXHandler();
parser->setDocumentHandler(handler);
parser->setErrorHandler(handler);
thread_vector.emplace_back(std::thread(parseFile,i,outputDir,handler,parser));
// parseFile(i,outputDir,handler,parser);
}
std::for_each(thread_vector.begin(),thread_vector.end(),std::mem_fn(&std::thread::join));
Inside the handle function I output some of the parsed items to an output file using std::ofstream and also use XMLString::transcode() to transcode XML strings. Although the result of the code running in single thread and multi-thread is similar, I find errors when I'm running it on valgrind with --tool=drd. Here is some of the error
==1045== Thread 7:
==1045== Conflicting load by thread 7 at 0x00126098 size 8
==1045== at 0x4D5E6E3: std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==1045== by 0x111DA8: parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x112A77: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*> > >::_M_run() (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x4D01DE3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==1045== by 0x48424BA: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_drd-amd64-linux.so)
==1045== by 0x4E30608: start_thread (pthread_create.c:477)
==1045== by 0x4F6A132: clone (clone.S:95)
==1045== Allocation context: BSS section of /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser
==1045== Other segment start (thread 2)
==1045== at 0x4F6A125: clone (clone.S:78)
==1045== Other segment end (thread 2)
==1045== at 0x484687C: pthread_mutex_unlock (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_drd-amd64-linux.so)
==1045== by 0x49D48AF: xercesc_3_2::XMLMutexLock::~XMLMutexLock() (Mutexes.cpp:84)
==1045== by 0x4B26C76: xercesc_3_2::ICULCPTranscoder::transcode(char16_t const*, xercesc_3_2::MemoryManager*) (ICUTransService.cpp:949)
==1045== by 0x4B2810E: xercesc_3_2::PosixFileMgr::fileOpen(char16_t const*, bool, xercesc_3_2::MemoryManager*) (PosixFileMgr.cpp:59)
==1045== by 0x49D2D8F: xercesc_3_2::BinFileInputStream::BinFileInputStream(char16_t const*, xercesc_3_2::MemoryManager*) (BinFileInputStream.cpp:40)
==1045== by 0x4A2C2E2: xercesc_3_2::LocalFileInputSource::makeStream() const (LocalFileInputSource.cpp:162)
==1045== by 0x4A5C237: xercesc_3_2::ReaderMgr::createReader(xercesc_3_2::InputSource const&, bool, xercesc_3_2::XMLReader::RefFrom, xercesc_3_2::XMLReader::Types, xercesc_3_2::XMLReader::Sources, bool, unsigned long) (ReaderMgr.cpp:379)
==1045== by 0x4A586CE: xercesc_3_2::IGXMLScanner::scanReset(xercesc_3_2::InputSource const&) (IGXMLScanner2.cpp:1287)
==1045== by 0x4A79C0D: xercesc_3_2::XMLScanner::scanFirst(xercesc_3_2::InputSource const&, xercesc_3_2::XMLPScanToken&) (XMLScanner.cpp:538)
==1045== by 0x4A7AE49: xercesc_3_2::XMLScanner::scanFirst(char16_t const*, xercesc_3_2::XMLPScanToken&) (XMLScanner.cpp:514)
==1045== by 0x4A7B1C9: xercesc_3_2::XMLScanner::scanFirst(char const*, xercesc_3_2::XMLPScanToken&) (XMLScanner.cpp:523)
==1045== by 0x111B65: parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== Other segment start (thread 3)
==1045== at 0x4F6A125: clone (clone.S:78)
==1045== Other segment end (thread 3)
==1045== at 0x484687C: pthread_mutex_unlock (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_drd-amd64-linux.so)
==1045== by 0x49D48AF: xercesc_3_2::XMLMutexLock::~XMLMutexLock() (Mutexes.cpp:84)
==1045== by 0x4B26C76: xercesc_3_2::ICULCPTranscoder::transcode(char16_t const*, xercesc_3_2::MemoryManager*) (ICUTransService.cpp:949)
==1045== by 0x11439B: SAXHandler::transcode[abi:cxx11](char16_t const*) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x11F38D: SAXHandler::endElement(char16_t const*) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x4AA4BC4: xercesc_3_2::SAXParser::endElement(xercesc_3_2::XMLElementDecl const&, unsigned int, bool, char16_t const*) (SAXParser.cpp:865)
==1045== by 0x4A4C588: xercesc_3_2::IGXMLScanner::scanEndTag(bool&) (IGXMLScanner.cpp:1178)
==1045== by 0x4A509FB: xercesc_3_2::IGXMLScanner::scanNext(xercesc_3_2::XMLPScanToken&) (IGXMLScanner.cpp:377)
==1045== by 0x111B86: parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x112A77: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*> > >::_M_run() (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x4D01DE3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==1045== by 0x48424BA: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_drd-amd64-linux.so)
==1045== Other segment start (thread 4)
==1045== at 0x4F6A125: clone (clone.S:78)
==1045== Other segment end (thread 4)
==1045== at 0x484687C: pthread_mutex_unlock (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_drd-amd64-linux.so)
==1045== by 0x49D48AF: xercesc_3_2::XMLMutexLock::~XMLMutexLock() (Mutexes.cpp:84)
==1045== by 0x4B26C76: xercesc_3_2::ICULCPTranscoder::transcode(char16_t const*, xercesc_3_2::MemoryManager*) (ICUTransService.cpp:949)
==1045== by 0x11439B: SAXHandler::transcode[abi:cxx11](char16_t const*) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x1173D0: SAXHandler::startElement(char16_t const*, xercesc_3_2::AttributeList&) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x4AA48C8: xercesc_3_2::SAXParser::startElement(xercesc_3_2::XMLElementDecl const&, unsigned int, char16_t const*, xercesc_3_2::RefVectorOf<xercesc_3_2::XMLAttr> const&, unsigned long, bool, bool) (SAXParser.cpp:982)
==1045== by 0x4A4EED7: xercesc_3_2::IGXMLScanner::scanStartTagNS(bool&) (IGXMLScanner.cpp:2632)
==1045== by 0x4A509DA: xercesc_3_2::IGXMLScanner::scanNext(xercesc_3_2::XMLPScanToken&) (IGXMLScanner.cpp:386)
==1045== by 0x111B86: parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x112A77: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*> > >::_M_run() (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x4D01DE3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==1045== by 0x48424BA: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_drd-amd64-linux.so)
==1045== Other segment start (thread 5)
==1045== at 0x4F6A125: clone (clone.S:78)
==1045== Other segment end (thread 5)
==1045== at 0x484687C: pthread_mutex_unlock (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_drd-amd64-linux.so)
==1045== by 0x49D48AF: xercesc_3_2::XMLMutexLock::~XMLMutexLock() (Mutexes.cpp:84)
==1045== by 0x4B26C76: xercesc_3_2::ICULCPTranscoder::transcode(char16_t const*, xercesc_3_2::MemoryManager*) (ICUTransService.cpp:949)
==1045== by 0x4B2810E: xercesc_3_2::PosixFileMgr::fileOpen(char16_t const*, bool, xercesc_3_2::MemoryManager*) (PosixFileMgr.cpp:59)
==1045== by 0x49D2D8F: xercesc_3_2::BinFileInputStream::BinFileInputStream(char16_t const*, xercesc_3_2::MemoryManager*) (BinFileInputStream.cpp:40)
==1045== by 0x4A2C2E2: xercesc_3_2::LocalFileInputSource::makeStream() const (LocalFileInputSource.cpp:162)
==1045== by 0x4A5C237: xercesc_3_2::ReaderMgr::createReader(xercesc_3_2::InputSource const&, bool, xercesc_3_2::XMLReader::RefFrom, xercesc_3_2::XMLReader::Types, xercesc_3_2::XMLReader::Sources, bool, unsigned long) (ReaderMgr.cpp:379)
==1045== by 0x4A586CE: xercesc_3_2::IGXMLScanner::scanReset(xercesc_3_2::InputSource const&) (IGXMLScanner2.cpp:1287)
==1045== by 0x4A79C0D: xercesc_3_2::XMLScanner::scanFirst(xercesc_3_2::InputSource const&, xercesc_3_2::XMLPScanToken&) (XMLScanner.cpp:538)
==1045== by 0x4A7AE49: xercesc_3_2::XMLScanner::scanFirst(char16_t const*, xercesc_3_2::XMLPScanToken&) (XMLScanner.cpp:514)
==1045== by 0x4A7B1C9: xercesc_3_2::XMLScanner::scanFirst(char const*, xercesc_3_2::XMLPScanToken&) (XMLScanner.cpp:523)
==1045== by 0x111B65: parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== Other segment start (thread 6)
==1045== at 0x4F6A125: clone (clone.S:78)
==1045== Other segment end (thread 6)
==1045== at 0x4F590AF: __libc_write (write.c:26)
==1045== by 0x4F590AF: write (write.c:24)
==1045== by 0x4ED9E8C: _IO_file_write##GLIBC_2.2.5 (fileops.c:1181)
==1045== by 0x4EDB950: new_do_write (fileops.c:449)
==1045== by 0x4EDB950: _IO_new_do_write (fileops.c:426)
==1045== by 0x4EDB950: _IO_do_write##GLIBC_2.2.5 (fileops.c:423)
==1045== by 0x4EDBE92: _IO_file_overflow##GLIBC_2.2.5 (fileops.c:784)
==1045== by 0x4ED6972: putc (putc.c:31)
==1045== by 0x4D5E258: std::ostream::put(char) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==1045== by 0x111B0D: parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x112A77: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*> > >::_M_run() (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x4D01DE3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==1045== by 0x48424BA: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_drd-amd64-linux.so)
==1045== by 0x4E30608: start_thread (pthread_create.c:477)
==1045== by 0x4F6A132: clone (clone.S:95)
==1045== Other segment start (thread 6)
==1045== at 0x4F6A125: clone (clone.S:78)
==1045== Other segment end (thread 6)
==1045== at 0x4F590AF: __libc_write (write.c:26)
==1045== by 0x4F590AF: write (write.c:24)
==1045== by 0x4ED9E8C: _IO_file_write##GLIBC_2.2.5 (fileops.c:1181)
==1045== by 0x4EDB950: new_do_write (fileops.c:449)
==1045== by 0x4EDB950: _IO_new_do_write (fileops.c:426)
==1045== by 0x4EDB950: _IO_do_write##GLIBC_2.2.5 (fileops.c:423)
==1045== by 0x4EDBE92: _IO_file_overflow##GLIBC_2.2.5 (fileops.c:784)
==1045== by 0x4ED6972: putc (putc.c:31)
==1045== by 0x4D5E258: std::ostream::put(char) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==1045== by 0x111B0D: parseFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*) (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x112A77: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, SAXHandler*, xercesc_3_2::SAXParser*> > >::_M_run() (in /mnt/c/Users/dennnn/Documents/Projects/ParserCPP/parser)
==1045== by 0x4D01DE3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==1045== by 0x48424BA: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_drd-amd64-linux.so)
==1045== by 0x4E30608: start_thread (pthread_create.c:477)
==1045== by 0x4F6A132: clone (clone.S:95)
I wonder why an object that is declared inside a threaded object can still be accessed by threads from another object (which I thought should be independent from each other and in this case is the XMLString::transcode() and std::ofstream) and cause data race as well as how to approach this problem. Thank you.
UPDATE 24/06/2022
The ICULCPTranscoder::transcode or the object used to transcode the XMLCh* is declared in the XMLPlatformUtils::Initialize() and is shared among the threads so it will use mutex lock to operate. Also, it might be that the conflicting objects belong to the same cache line.
I'm linking a library against opencv libraries. I noticed that when I run the unit test executable (using GTest) for that library, that valgrind reports a memory leak/conditional jump or move depends on uninitialised value, even when the test doesn't do anything at all (just empty test method).
I traced that issue back to a single line in my CMakeLists.txt - when I remove the linkage against opencv, the memory leak goes away. This is the releavant snippet:
find_package(OpenCV REQUIRED)
set(libImageSources src/SourceImageFile.cpp)
add_library(image SHARED ${libImageSources})
# removing this line fixes the leak:
target_link_libraries(image ${OpenCV_LIBS})
Otherwise when runnig an empty unit test, I get an error like this:
==18681== Memcheck, a memory error detector
==18681== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==18681== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==18681== Command: ./image_test /Users/max/Documents/playground/cpp/image-server/tests
==18681==
==18681== Conditional jump or move depends on uninitialised value(s)
==18681== at 0x7FFF5FC24A27: bcmp (in /usr/lib/dyld)
==18681== by 0x7FFF5FC11904: ImageLoaderMachO::validateFirstPages(linkedit_data_command const*, int, unsigned char const*, unsigned long, long long, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC16B7A: ImageLoaderMachOCompressed::instantiateFromFile(char const*, int, unsigned char const*, unsigned long, unsigned long long, unsigned long long, stat const&, unsigned int, unsigned int, linkedit_data_command const*, encryption_info_command const*, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC10A6E: ImageLoaderMachO::instantiateFromFile(char const*, int, unsigned char const*, unsigned long long, unsigned long long, stat const&, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC038C2: dyld::loadPhase6(int, stat const&, char const*, dyld::LoadContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08468: dyld::loadPhase5(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08188: dyld::loadPhase4(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC07EED: dyld::loadPhase3(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC07642: dyld::loadPhase1(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC0347A: dyld::loadPhase0(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC0315E: dyld::load(char const*, dyld::LoadContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08705: dyld::libraryLocator(char const*, bool, char const*, ImageLoader::RPathChain const*) (in /usr/lib/dyld)
==18681==
==18681== Use of uninitialised value of size 8
==18681== at 0x7FFF5FC24A3F: bcmp (in /usr/lib/dyld)
==18681== by 0x7FFF5FC11904: ImageLoaderMachO::validateFirstPages(linkedit_data_command const*, int, unsigned char const*, unsigned long, long long, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC16B7A: ImageLoaderMachOCompressed::instantiateFromFile(char const*, int, unsigned char const*, unsigned long, unsigned long long, unsigned long long, stat const&, unsigned int, unsigned int, linkedit_data_command const*, encryption_info_command const*, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC10A6E: ImageLoaderMachO::instantiateFromFile(char const*, int, unsigned char const*, unsigned long long, unsigned long long, stat const&, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC038C2: dyld::loadPhase6(int, stat const&, char const*, dyld::LoadContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08468: dyld::loadPhase5(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08188: dyld::loadPhase4(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC07EED: dyld::loadPhase3(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC07642: dyld::loadPhase1(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC0347A: dyld::loadPhase0(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC0315E: dyld::load(char const*, dyld::LoadContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08705: dyld::libraryLocator(char const*, bool, char const*, ImageLoader::RPathChain const*) (in /usr/lib/dyld)
==18681==
==18681== Use of uninitialised value of size 8
==18681== at 0x7FFF5FC24A44: bcmp (in /usr/lib/dyld)
==18681== by 0x7FFF5FC11904: ImageLoaderMachO::validateFirstPages(linkedit_data_command const*, int, unsigned char const*, unsigned long, long long, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC16B7A: ImageLoaderMachOCompressed::instantiateFromFile(char const*, int, unsigned char const*, unsigned long, unsigned long long, unsigned long long, stat const&, unsigned int, unsigned int, linkedit_data_command const*, encryption_info_command const*, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC10A6E: ImageLoaderMachO::instantiateFromFile(char const*, int, unsigned char const*, unsigned long long, unsigned long long, stat const&, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC038C2: dyld::loadPhase6(int, stat const&, char const*, dyld::LoadContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08468: dyld::loadPhase5(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08188: dyld::loadPhase4(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC07EED: dyld::loadPhase3(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC07642: dyld::loadPhase1(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC0347A: dyld::loadPhase0(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC0315E: dyld::load(char const*, dyld::LoadContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08705: dyld::libraryLocator(char const*, bool, char const*, ImageLoader::RPathChain const*) (in /usr/lib/dyld)
==18681==
==18681== Conditional jump or move depends on uninitialised value(s)
==18681== at 0x7FFF5FC11907: ImageLoaderMachO::validateFirstPages(linkedit_data_command const*, int, unsigned char const*, unsigned long, long long, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC16B7A: ImageLoaderMachOCompressed::instantiateFromFile(char const*, int, unsigned char const*, unsigned long, unsigned long long, unsigned long long, stat const&, unsigned int, unsigned int, linkedit_data_command const*, encryption_info_command const*, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC10A6E: ImageLoaderMachO::instantiateFromFile(char const*, int, unsigned char const*, unsigned long long, unsigned long long, stat const&, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC038C2: dyld::loadPhase6(int, stat const&, char const*, dyld::LoadContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08468: dyld::loadPhase5(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08188: dyld::loadPhase4(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC07EED: dyld::loadPhase3(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC07642: dyld::loadPhase1(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC0347A: dyld::loadPhase0(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC0315E: dyld::load(char const*, dyld::LoadContext const&) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC08705: dyld::libraryLocator(char const*, bool, char const*, ImageLoader::RPathChain const*) (in /usr/lib/dyld)
==18681== by 0x7FFF5FC0E85D: ImageLoader::recursiveLoadLibraries(ImageLoader::LinkContext const&, bool, ImageLoader::RPathChain const&) (in /usr/lib/dyld)
==18681==
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from ImageProcessingConfigurationService
[ RUN ] ImageProcessingConfigurationService.evaluateConfigurationFile
[ OK ] ImageProcessingConfigurationService.evaluateConfigurationFile (13 ms)
[----------] 1 test from ImageProcessingConfigurationService (28 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (114 ms total)
[ PASSED ] 1 test.
==18681==
==18681== HEAP SUMMARY:
==18681== in use at exit: 1,346,030 bytes in 1,453 blocks
==18681== total heap usage: 2,926 allocs, 1,473 frees, 3,045,052 bytes allocated
==18681==
==18681== LEAK SUMMARY:
==18681== definitely lost: 10,580 bytes in 137 blocks
==18681== indirectly lost: 10,032 bytes in 152 blocks
==18681== possibly lost: 9,568 bytes in 183 blocks
==18681== still reachable: 43,598 bytes in 520 blocks
==18681== suppressed: 1,272,252 bytes in 461 blocks
==18681== Rerun with --leak-check=full to see details of leaked memory
==18681==
==18681== For counts of detected and suppressed errors, rerun with: -v
==18681== Use --track-origins=yes to see where uninitialised values come from
==18681== ERROR SUMMARY: 36 errors from 4 contexts (suppressed: 0 from 0)
Compiler is clang++ Apple LLVM version 7.3.0 (clang-703.0.31), OS X El Capitan (10.11.1).
Anyone else has had an issue like that? It seems stuff works nontheless, but the error creates noise that I'd rather ignore, I'm not sure if I can fix the root cause.
The issue seems like a false positive reported by valgrind which I can ignore as it's not caused by code.
I tried (at the time of writing) the latest valgrind suppressions file for macOS which didn't work for me, the issue was still reported.
I let valgrind dump the suppressions needed to ignore this error, using a command like:
valgrind --leak-check=full --show-reachable=yes --error-limit=no --gen-suppressions=all --log-file=l.log ./image_test
And then used a tool to parse the log output (the awk script didn't work for me as it contained a syntax error, I used the perl version) into a suppressions file:
cat l.log | ./parse_valgrind_suppressions.pl > s.supp
Now running the test with the generated suppresions file ignores the memory leak:
valgrind --suppressions=./s.supp ./image_test
I am using valkrie/valgrind on Mac OS 10.8 to find memory leaks in a small testcase command line tool.
While I have been able to locate many problems successfully before, valgrind right now presents the following error to me of which I can't seem to make sense:
==69195== Conditional jump or move depends on uninitialised value(s)
==69195== at 0x3F68CF: ??? (in /usr/lib/system/libxpc.dylib)
==69195== by 0x11AC2F: ??? (in /usr/lib/libSystem.B.dylib)
==69195== by 0x8FE13CD9: ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==69195== by 0x8FE13FDD: ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==69195== by 0x8FE10267: ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&) (in /usr/lib/dyld)
==69195== by 0x8FE101CB: ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&) (in /usr/lib/dyld)
==69195== by 0x8FE100B9: ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) (in /usr/lib/dyld)
==69195== by 0x8FE01DD8: dyld::initializeMainExecutable() (in /usr/lib/dyld)
==69195== by 0x8FE05ADA: dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) (in /usr/lib/dyld)
==69195== by 0x8FE01375: dyldbootstrap::start(macho_header const*, int, char const**, long, macho_header const*, unsigned long*) (in /usr/lib/dyld)
==69195== by 0x8FE01076: _dyld_start (in /usr/lib/dyld)
Usually valgrind's error traces touch my own code before they descend into library/system calls,
but in this case the error seems to occur in "dyld::initializeMainExecutable()", which is in a system linker library outside of my control.
Where could I look to fix this - is there another way to make sense of it?
Cheers+thanks y
I am new to valgrind, so this could be missintepreting the output.
Valgrind seems to think bytes are being lost in the following lines:
if ( mysqlpp::StoreQueryResult result = query.store() ){
for ( size_t i = 0; i < result.num_rows(); ++i ){
hash = std::string( result[i]["hash"].data(), result[i]["hash"].length() );
label = std::string( result[i]["label"].data(), result[i]["label"].length() );
keywords = std::string( result[i]["keywords"].data(), result[i]["keywords"].length() );
task_raw = std::string( result[i]["task"].data(), result[i]["task"].length() );
solution_raw = std::string( result[i]["solution"].data(), result[i]["solution"].length() );
In fact it complains every time there is an std::string being constructed this way^^. I cannot figure out if this is a false-positive, if the way I'm copying is not the right way, or if the result is being leaked somehow.
Interestingly enough, it complains about another piece of code that doesn't use mysql++, but again a string constructor is used:
for ( auto & f : frames ){
for ( unsigned int i = 0; i < f.getTask().size(); i++ ){
if ( i + 1 < f.getTask().size() ){
auto t = f.getTask()[i].value();
auto n = f.getTask()[i+1].value();
pairs.push_back( std::string( t + n ) );
The line where a string is being constructed and pushed back, are possibly lost. Then a detailed output of malloc, operator new, std::string, etc are given.
The same story appears again, when copying a string from a std::istringstream:
std::copy( std::istream_iterator<std::string>(iss), std::istream_iterator<std::string>(), std::back_inserter<std::vector<std::string> >(tokens));
Are those false-positives? Does it have to do with using C++0x? Or are those legit leaks that may be originating from somewhere I have not yet detected to be leaking?
EDIT
Below is a sample of the reoccurring valgrind warning:
==18141== 9,555 bytes in 147 blocks are possibly lost in loss record 250 of 256
==18141== at 0x4C28BED: malloc (vg_replace_malloc.c:263)
==18141== by 0x7CA74C: operator new(unsigned long) (in trunk/Build/icarus)
==18141== by 0x7B4208: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (in /trunk/Build/icarus)
==18141== by 0x7B5904: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (in trunk/Build/icarus)
==18141== by 0x7B5A7C: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&) (in trunk/Build/icarus)
==18141== by 0x610D65: Frame::mySQLloadFrameFromExample(std::string const&) (Frame.cpp:80)
==18141== by 0x6103FB: Frame::Frame(std::string const&) (Frame.cpp:5)
==18141== by 0x6054DB: Domain::mySQLloadFrames() (Domain.cpp:90)
==18141== by 0x60489D: Domain::Domain(std::string) (Domain.cpp:7)
==18141== by 0x628876: DomainController::loadDomainData() (DomainController.cpp:33)
==18141== by 0x628582: DomainController::DomainController() (DomainController.cpp:5)
==18141== by 0x627002: Controller::Controller() (Controller.cpp:26)
They are almost all identical, with the only difference being the actual line in code (i.e: the string constructor line).
The second type of warning is:
==18141== 3,562 bytes in 137 blocks are possibly lost in loss record 242 of 256
==18141== at 0x4C28BED: malloc (vg_replace_malloc.c:263)
==18141== by 0x7CA74C: operator new(unsigned long) (in trunk/Build/icarus)
==18141== by 0x7B4208: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (in trunk/Build/icarus)
==18141== by 0x7B4EFA: std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long) (in trunk/Build/icarus)
==18141== by 0x7B4FDF: std::string::reserve(unsigned long) (in trunk/Build/icarus)
==18141== by 0x784F2F: std::basic_istream<char, std::char_traits<char> >& std::operator>><char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (in trunk/Build/icarus)
==18141== by 0x61208E: std::istream_iterator<std::string, char, std::char_traits<char>, long>::_M_read() (stream_iterator.h:121)
==18141== by 0x613405: std::istream_iterator<std::string, char, std::char_traits<char>, long>::operator++() (stream_iterator.h:95)
==18141== by 0x6131BC: std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > > std::__copy_move<false, false, std::input_iterator_tag>::__copy_m<std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > > >(std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > >) (stl_algobase.h:298)
==18141== by 0x612D09: std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > > std::__copy_move_a<false, std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > > >(std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > >) (stl_algobase.h:384)
==18141== by 0x6121FD: std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > > std::__copy_move_a2<false, std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > > >(std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > >) (stl_algobase.h:422)
==18141== by 0x611C6A: std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > > std::copy<std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > > >(std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::istream_iterator<std::string, char, std::char_traits<char>, long>, std::back_insert_iterator<std::vector<std::string, std::allocator<std::string> > >) (stl_algobase.h:454)
Which seems to be related with the std::istringstream code mentioned above.
I seem to have a corrupted heap, I cannot figure out why this is happening....
Following is the trace from valgrind..
==12697== Use of uninitialised value of size 4
==12697== at 0xDD0725: __gnu_cxx::__atomic_add(int volatile*, int) (in /usr/lib/libstdc++.so.6.0.7)
==12697== by 0x1C3AD9BB: chargeRate::chargeRate(chargeRate const&) (in /root//app/libapp++.so)
==12697== by 0x1C4C9C22: __gnu_cxx::__mt_alloc_base<chargeRate>::construct(chargeRate*, chargeRate const&) (mt_allocator.h:585)
==12697== by 0x1C4CAC9F: std::vector<chargeRate, std::allocator<chargeRate> >::_M_insert_aux(__gnu_cxx::__normal_iterator<chargeRate*, std::vector<chargeRate, std::allocator<chargeRate> > >, chargeRate const&) (vector.tcc:284)
==12697== by 0x1C4CAF9E: std::vector<chargeRate, std::allocator<chargeRate> >::push_back(chargeRate const&) (stl_vector.h:610)
==12697== by 0x1C4C8A03: WebTranslations::getChargeDetails(std::vector<std::string, std::allocator<std::string> >&, std::vector<chargeRate, std::allocator<chargeRate> >&) (WebTranslations.cpp:427)
==12697== by 0x1C4C8F83: WebTranslations::getChargeTranslations(std::vector<std::string, std::allocator<std::string> >&, std::vector<std::string, std::allocator<std::string> >&) (WebTranslations.cpp:1172)
==12697==
==12697== Invalid read of size 4
==12697== at 0xDB468B: std::string::string(std::string const&) (in /usr/lib/libstdc++.so.6.0.7)
==12697== by 0x1C3AD9E0: chargeRate::chargeRate(chargeRate const&) (in /root//app/libapp++.so)
==12697== by 0x1C4C9C22: __gnu_cxx::__mt_alloc_base<chargeRate>::construct(chargeRate*, chargeRate const&) (mt_allocator.h:585)
==12697== by 0x1C4CAC9F: std::vector<chargeRate, std::allocator<chargeRate> >::_M_insert_aux(__gnu_cxx::__normal_iterator<chargeRate*, std::vector<chargeRate, std::allocator<chargeRate> > >, chargeRate const&) (vector.tcc:284)
==12697== by 0x1C4CAF9E: std::vector<chargeRate, std::allocator<chargeRate> >::push_back(chargeRate const&) (stl_vector.h:610)
==12697== by 0x1C4C8A03: WebTranslations::getChargeDetails(std::vector<std::string, std::allocator<std::string> >&, std::vector<chargeRate, std::allocator<chargeRate> >&) (WebTranslations.cpp:427)
==12697== by 0x1C4C8F83: WebTranslations::getChargeTranslations(std::vector<std::string, std::allocator<std::string> >&, std::vector<std::string, std::allocator<std::string> >&) (WebTranslations.cpp:1172)
==12697== by 0x1C299E48: zif__get_charge_translations (in /usr/lib/php4/.so)
==12697== by 0x1BCE0916: zend_do_fcall_common_helper (in /usr/lib/httpd/modules/libphp5.so)
==12697== by 0x1BCF1088: zend_do_fcall_handler (in /usr/lib/httpd/modules/libphp5.so)
==12697== by 0x1BCDDD92: execute (in /usr/lib/httpd/modules/libphp5.so)
==12697== by 0x1BCE02A9: zend_do_fcall_common_helper (in /usr/lib/httpd/modules/libphp5.so)
==12697== Address 0xFFFFFFFC is not stack'd, malloc'd or (recently) free'd
==12697==
==12697== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==12697== GPF (Pointer out of bounds?)
==12697== at 0xDB468B: std::string::string(std::string const&) (in /usr/lib/libstdc++.so.6.0.7)
==12697== by 0x1C3AD9E0: chargeRate::chargeRate(chargeRate const&) (in /root//app/libapp++.so)
==12697== by 0x1C4C9C22: __gnu_cxx::__mt_alloc_base<chargeRate>::construct(chargeRate*, chargeRate const&) (mt_allocator.h:585)
==12697== by 0x1C4CAC9F: std::vector<chargeRate, std::allocator<chargeRate> >::_M_insert_aux(__gnu_cxx::__normal_iterator<chargeRate*, std::vector<chargeRate, std::allocator<chargeRate> > >, chargeRate const&) (vector.tcc:284)
==12697== by 0x1C4CAF9E: std::vector<chargeRate, std::allocator<chargeRate> >::push_back(chargeRate const&) (stl_vector.h:610)
==12697== by 0x1C4C8A03: WebTranslations::getChargeDetails(std::vector<std::string, std::allocator<std::string> >&, std::vector<chargeRate, std::allocator<chargeRate> >&) (WebTranslations.cpp:427)
==12697== by 0x1C4C8F83: WebTranslations::getChargeTranslations(std::vector<std::string, std::allocator<std::string> >&, std::vector<std::string, std::allocator<std::string> >&) (WebTranslations.cpp:1172)
The code is just pushing a structure into a vector. The structure has a few string variables in it. Please let me know if the source is required for further debugging.
Some source:
typedef struct{
string chargeType; // The name of the charge type, eg "Date Units"
string unitSize;
string rate;
bool perConnection;
string cap;
bool useMaxDailyCharge;
string maxDailyCharge;
string identifier;
} chargeRate;
getChargeDetails :
vector<chargeRate> my_vector;
my_vector.push_back(this->getChargeRateDetails(chargeStructureNames[i]));
getChargeRateDetails :
where : vector<vector<string> > StringMatrix
StringMatrix *results; //used to retrive results from database.
chargeRate chargeInformation;
...
//populate results, check them
..
chargeInformation.chargeType = (*results)[FIRST_ROW][CHARGE_TYPE];
return chargeInformation;
Edit : I am aware this is returning a "copy" .. this is done test out a few things, i am inserting the same into a copy of vector passed by reference.
Cheers!
In case getChargeRateDetails returns by reference you are trying to use a reference to a variable that is already out of scope.
It looks like your copy constructor doesn't do the right thing. In order to push an element onto a vector, a copy of your chargeRate is created. At first sight,
at 0xDD0725: __gnu_cxx::__atomic_add(int volatile*, int)
chargeRate::chargeRate(chargeRate const&)
Tells me that the chargeRate copy constructor tries to add something to an uninitialized variable (pointed at by the int volatile* argument of __atomic_add.
Probably, you are __atomic_adding something to an uninitialized member variable.
StringMatrix *results;
Seems intended to refer to an array of strings. Chances are that the StringMatrix is not initialized well. Hence, the first row may contain an uninitialized string-like piece of memory, that is used to copy-construct the chargeType member of your new object.
The string class contains a reference counter that needs to be increased when a new string points to the same data, so this may make sense.
You can verify this by (temporarily) assigning e.g. an empty string instead of (*results)[FIRST_ROW][CHARGE_TYPE].
Not sure what is going on in your application, but maybe it has to do with (not) using back_inserter?
there is a good example here.