I have a C++ app and I know in advance that I am expecting around 1000 elements (sometimes few more) to store and process for hundreds of instances.
The issue is that the memory used by the process increases slowly but the memory of the server is overwhelmed in few hours.
I am using valgrind for detecting eventual memory leaks.
Edit
First thanks to all of you for the suggestions. I managed to solve many errors. I added virtual destructors to some classes and changed some pointers to boost::shared_ptr using reset to delete them. However I still have an increasing memory usage over the time.
The main parts of the app are as follows :
#ifdef GLOBAL
#define Global
#else
#define Global extern
#endif
typedef std::map<uint32_t, ResultStruct> ClassDResultMap ;
typedef std::map<uint32_t, ClassDResultMap > ClassCResultMap ;
typedef std::map<uint32_t, ClassCResultMap > ClassBResultMap ;
typedef std::map<uint32_t, ClassBResultMap > ClassAResultMap ;
Global ClassAResultMap classAresultmap;
ClassZero creates many ClassA
typedef std::map<uint32_t,ClassA*> classAMap;
classA_[aid] = new ClassA(io_service_, filename.c_str(), id, pri_queue_);
Inside ClassA, up to 3200 ClassB instance are created:
typedef std::map<uint32_t, boost::shared_ptr<ClassB >> classBMap ;
classBMap[aid].insert(std::pair<uint32_t, boost::shared_ptr<ClassB>>(bid, boost::make_shared<ClassB >(io_service_, pri_queue_, ...)));
Inside ClassB, up to 4 ClassC instances are created inside a ConfigMap struct having a member boost::shared_ptr<ClassC > classC_; :
typedef std::map<uint32_t, classCConfig> ConfigMap;
(ConfigMap::iterator)->second.classC_ = boost::make_shared<ClassC >(io_service_, pri_queue_, ...);
Inside ClassC, one ClassD is instantiated by ClassC constructor
ClassC::ClassC(...): ClassD(io_service_, pri_queue_,...),... { }
Then Inside ClassD, elements are processed and results are calculated every 10-30 seconds than stored in ClassDResultMap :
(ClassDResultMap::iterator)->second[results.stime] = results;
Finally, inside another class, ClassStorage, all the results in the global ClassAResultmap are saved into files and cleared :
std::ofstream res_stream_;
res_stream_ << "{ \"start\" : " << results_it->second.stime << ...;
(ClassDResultMap::iterator)->second.clear();
std::map<uint32_t, ClassDResultMap>().swap((ClassDResultMap::iterator)->second);
This what I get with Valgrind with 1 Class A : 8 ClassB : 32 class C/D
==9544== HEAP SUMMARY:
==9544== in use at exit: 25,817 bytes in 171 blocks
==9544== total heap usage: 148,771 allocs, 148,600 frees, 16,803,272 bytes allocated
==9544==
==9544== 1,216 bytes in 4 blocks are possibly lost in loss record 39 of 45
==9544== at 0x4C2677B: calloc (vg_replace_malloc.c:593)
==9544== by 0x40118A2: _dl_allocate_tls (in /lib64/ld-2.12.so)
==9544== by 0x5F971E8: pthread_create##GLIBC_2.2.5 (in /lib64/libpthread-2.12.so)
==9544== by 0x4E3ACC0: boost::thread::start_thread_noexcept() (in /usr/local/lib/libboost_thread.so.1.54.0)
==9544== by 0x45EE92: boost::thread::start_thread() (thread.hpp:180)
==9544== by 0x461429: boost::thread::thread<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_service>, boost::_bi::list1<boost::reference_wrapper<boost::asio::io_service> > >&>(boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_service>, boost::_bi::list1<boost::reference_wrapper<boost::asio::io_service> > >&) (thread.hpp:267)
==9544== by 0x4605F7: boost::thread* boost::thread_group::create_thread<boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_service>, boost::_bi::list1<boost::reference_wrapper<boost::asio::io_service> > > >(boost::_bi::bind_t<unsigned long, boost::_mfi::mf0<unsigned long, boost::asio::io_service>, boost::_bi::list1<boost::reference_wrapper<boost::asio::io_service> > >) (thread_group.hpp:78)
==9544== by 0x45CAC9: main (main.cpp:151)
==9544==
==9544== LEAK SUMMARY:
==9544== definitely lost: 0 bytes in 0 blocks
==9544== indirectly lost: 0 bytes in 0 blocks
==9544== possibly lost: 1,216 bytes in 4 blocks
==9544== still reachable: 24,601 bytes in 167 blocks
==9544== suppressed: 0 bytes in 0 blocks
==9544== Reachable blocks (those to which a pointer was found) are not shown.
==9544== To see them, rerun with: --leak-check=full --show-reachable=yes
==9544==
==9544== For counts of detected and suppressed errors, rerun with: -v
==9544== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6)
and the part of main.cpp which Valgrind complains about :
boost::asio::io_service work_io_service;
ClassZero manager(work_io_service);
boost::thread_group work_threads;
uint32_t workers = boost::thread::hardware_concurrency();
for (std::size_t i = 0; i != workers; ++i)
{
work_threads.create_thread(
boost::bind(&boost::asio::io_service::run, boost::ref(work_io_service)));
}
My issue is that in almost 20 minutes, the server memory (4GB ) increased from 3% to 17% and keeps going on. The process memory usage is increasing but stays below 4% !
Related
I'm a new C++ coder and met a problem.
Please view the following code as an example:
#include <iostream>
#include <vector>
#include <thread>
class student{
public:
std::string name;
int id;
};
void foo(){
int k = 1;
std::cout<<"thread a"<<std::endl;
return;
}
void foo2(){
int k = 1;
std::cout<<"thread b"<<std::endl;
return;
}
void foo3(){
std::thread a(foo);
std::thread b(foo2);
a.detach();
b.detach();
return;
}
int main(){
foo3();
return 0;
}
That's an example code. I want two threads running independently and release memory after each one ends. I use valgrind to check memory leak. It shows an error like this:
==63346== Memcheck, a memory error detector
==63346== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63346== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==63346== Command: ./a.out
==63346==
==63346==
==63346== HEAP SUMMARY:
==63346== in use at exit: 608 bytes in 4 blocks
==63346== total heap usage: 5 allocs, 1 frees, 73,312 bytes allocated
==63346==
==63346== 288 bytes in 1 blocks are possibly lost in loss record 3 of 4
==63346== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==63346== by 0x40147D9: calloc (rtld-malloc.h:44)
==63346== by 0x40147D9: allocate_dtv (dl-tls.c:375)
==63346== by 0x40147D9: _dl_allocate_tls (dl-tls.c:634)
==63346== by 0x4B4D834: allocate_stack (allocatestack.c:430)
==63346== by 0x4B4D834: pthread_create##GLIBC_2.34 (pthread_create.c:647)
==63346== by 0x494A388: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
==63346== by 0x1095B9: std::thread::thread<void (&)(), , void>(void (&)()) (in /home/alan/Avionics/test/a.out)
==63346== by 0x10935C: foo3() (in /home/alan/Avionics/test/a.out)
==63346== by 0x109400: main (in /home/alan/Avionics/test/a.out)
==63346==
==63346== 288 bytes in 1 blocks are possibly lost in loss record 4 of 4
==63346== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==63346== by 0x40147D9: calloc (rtld-malloc.h:44)
==63346== by 0x40147D9: allocate_dtv (dl-tls.c:375)
==63346== by 0x40147D9: _dl_allocate_tls (dl-tls.c:634)
==63346== by 0x4B4D834: allocate_stack (allocatestack.c:430)
==63346== by 0x4B4D834: pthread_create##GLIBC_2.34 (pthread_create.c:647)
==63346== by 0x494A388: std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
==63346== by 0x1095B9: std::thread::thread<void (&)(), , void>(void (&)()) (in /home/alan/Avionics/test/a.out)
==63346== by 0x109372: foo3() (in /home/alan/Avionics/test/a.out)
==63346== by 0x109400: main (in /home/alan/Avionics/test/a.out)
==63346==
==63346== LEAK SUMMARY:
==63346== definitely lost: 0 bytes in 0 blocks
==63346== indirectly lost: 0 bytes in 0 blocks
==63346== possibly lost: 576 bytes in 2 blocks
==63346== still reachable: 32 bytes in 2 blocks
==63346== suppressed: 0 bytes in 0 blocks
==63346== Reachable blocks (those to which a pointer was found) are not shown.
==63346== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==63346==
==63346== For lists of detected and suppressed errors, rerun with: -s
==63346== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
If i replace detach() by join(), there's no memory leak, I'm a little confused. Does anyone can give me some advice? Thanks a lot!
You need to make your main wait until the threads have finished running (or at least that they won't access any global objects after). If you don't make sure of that, not only will you have memory leaks, you may also have undefined behavior.
join does exactly make sure of that. The thread calling it will wait until the thread on which join is called finishes.
On the other hand detach is used to when you don't want to keep the std::thread object around to call join on it, while still having the thread function continue to run. If you detach a thread you need to use some other synchronization mechanism outside the join mechanism to make sure that main doesn't exit while the threads are still doing work (or at least accessing global objects).
There is rarely a need to detach a thread. Making it work is complicated. You don't need to do that.
Instead store the std::thread object somewhere and call join on it when you want to wait for your threads to end, at latest before main returns. That should be the default approach if you don't have a good reason to not use it. If you are a beginner, then you can forget about detach for now.
I'm learn about std::allocator. I try to allocate but use deallocate incorrectly I saw that it didn't use size argument, I confuse about the method could you please explain for me ? Thanks.
testcase1 "test" : I didn't deallocate, valgrind detected (correct)
testcase2 "test_deallocate" : I deallocate with size(0) less than actual size (400),valgrind or -fsanitize=address can't detect leak
testcase3 "test_deallocate2": I deallocate with size(10000) greater than actual size (400) compiler didn't warning , g++ with -fsanitize=address also can't detect this.
#include <iostream>
#include <memory>
using namespace std;
void test(){
allocator<int> al;
int* bl = al.allocate(100);
}
void test_deallocate(){
allocator<int> al;
int* bl = al.allocate(100);
al.deallocate(bl, 0);
}
void test_deallocate2(){
allocator<int> al;
int* bl = al.allocate(100);
al.deallocate(bl, 10000);
}
int main(){
test();
test_deallocate();
test_deallocate2();
return 0;
}
Valgrind:
valgrind --leak-check=full ./a.out
==12655== Memcheck, a memory error detector
==12655== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12655== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==12655== Command: ./a.out
==12655==
==12655==
==12655== HEAP SUMMARY:
==12655== in use at exit: 400 bytes in 1 blocks
==12655== total heap usage: 4 allocs, 3 frees, 73,904 bytes allocated
==12655==
==12655== 400 bytes in 1 blocks are definitely lost in loss record 1 of 1
==12655== at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==12655== by 0x1090D1: allocate (new_allocator.h:114)
==12655== by 0x1090D1: test (test.cpp:8)
==12655== by 0x1090D1: main (test.cpp:27)
==12655==
==12655== LEAK SUMMARY:
==12655== definitely lost: 400 bytes in 1 blocks
==12655== indirectly lost: 0 bytes in 0 blocks
==12655== possibly lost: 0 bytes in 0 blocks
==12655== still reachable: 0 bytes in 0 blocks
==12655== suppressed: 0 bytes in 0 blocks
==12655==
==12655== For lists of detected and suppressed errors, rerun with: -s
==12655== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Valgrind only intercepts the lower level allocation functions (malloc, new etc.). So it all depends on what the implementation of allocate and deallocate do.
Currently Valgrind doesn't do much checking of sized delete (or aligned new for that matter, see why does std::allocator::deallocate require a size? and https://bugs.kde.org/show_bug.cgi?id=433859).
I may implement something for these in 2022.
If you use GCC libstdc++ or clang libc++ then their sized deletes do not do anything special, they just call plain delete. I haven't looked for deallocate.
testcase2 "test_deallocate" : I deallocate with size(0) less than actual size (400),valgrind or -fsanitize=address can't detect leak
When you dellocate with wrong size, then the behaviour of the program is undefined. When the behaviour is undefined, there is no guarantee that memory would be leaked.
testcase3 "test_deallocate2": I deallocate with size(10000) greater than actual size (400) compiler didn't warning , g++ with -fsanitize=address also can't detect this.
Ditto.
I checked libstdc++ which used in the testcases.
I saw that currently size argument for std::allocator::deallocate is unused. it call delete, same with new/delete operator operation, it don't need size anymore.
00096 void
00097 deallocate(pointer __p, size_type)
00098 { ::operator delete(__p); }
https://gcc.gnu.org/onlinedocs/gcc-4.6.2/libstdc++/api/a00958_source.html
I need to overwrite an element of an Rcpp::List object passed as parameter to an Rcpp function. My concern is memory safety. Is it the case that by reassigning a non-empty element of the list, I am effectively rewiring a pointer to the original content, yet never deallocating the memory which stores the original content? If it is, how does one solve this?
I am aware that I can easily modify an Rcpp object (eg. Rcpp::NumericVector) that is an element of an Rcpp::List, since Rcpp::NumericVector makes a shallow copy. This does not satisfy my requirement, however, which is to replace the element entirely by something else.
Below, I include a C++ code snippet which shows the scenario to which I am referring.
#include <Rcpp.h>
using namespace Rcpp;
// [[Rcpp::export]]
void replaceListElement(List l)
{
std::vector<int> v;
v.push_back(4);
v.push_back(5);
v.push_back(6);
l["a"] = v;
}
/*** R
l <- list()
l$a <- c(1,2,3)
replaceListElement(l)
print(l)
*/
When sourced via Rcpp in RStudio, the print(l) command outputs the following
$a
[1] 4 5 6
which is the desired result, so my question pertains only to memory safety.
A Rcpp::List is a Vector<VECSXP>, i.e. a vector of pointers to other vectors. If you assign a new vector to some element in this list, you are indeed just changing a pointer without freeing the memory that the pointer used to point to. However, R still knows about this memory and frees it through its garbage collector. We can see this in action with a simple experiment, in which I use your C++ code with a slight change in the R code:
#include <Rcpp.h>
using namespace Rcpp;
// [[Rcpp::export]]
void replaceListElement(List l)
{
std::vector<int> v;
v.push_back(4);
v.push_back(5);
v.push_back(6);
l["a"] = v;
}
/*** R
l <- list()
l$a <- runif(1e7)
replaceListElement(l)
print(l)
gc() # optional
*/
Here a larger vector is used to make the effect more prominent. If I now use R -d valgrind -e 'Rcpp::sourceCpp("<filename>")' I get the following result with the gc() call
==13827==
==13827== HEAP SUMMARY:
==13827== in use at exit: 48,125,775 bytes in 9,425 blocks
==13827== total heap usage: 34,139 allocs, 24,714 frees, 173,261,724 bytes allocated
==13827==
==13827== LEAK SUMMARY:
==13827== definitely lost: 0 bytes in 0 blocks
==13827== indirectly lost: 0 bytes in 0 blocks
==13827== possibly lost: 0 bytes in 0 blocks
==13827== still reachable: 48,125,775 bytes in 9,425 blocks
==13827== of which reachable via heuristic:
==13827== newarray : 4,264 bytes in 1 blocks
==13827== suppressed: 0 bytes in 0 blocks
==13827== Rerun with --leak-check=full to see details of leaked memory
==13827==
==13827== For counts of detected and suppressed errors, rerun with: -v
==13827== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
And without the gc() call:
==13761==
==13761== HEAP SUMMARY:
==13761== in use at exit: 132,713,314 bytes in 10,009 blocks
==13761== total heap usage: 34,086 allocs, 24,077 frees, 173,212,886 bytes allocated
==13761==
==13761== LEAK SUMMARY:
==13761== definitely lost: 0 bytes in 0 blocks
==13761== indirectly lost: 0 bytes in 0 blocks
==13761== possibly lost: 0 bytes in 0 blocks
==13761== still reachable: 132,713,314 bytes in 10,009 blocks
==13761== of which reachable via heuristic:
==13761== newarray : 4,264 bytes in 1 blocks
==13761== suppressed: 0 bytes in 0 blocks
==13761== Rerun with --leak-check=full to see details of leaked memory
==13761==
==13761== For counts of detected and suppressed errors, rerun with: -v
==13761== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
So in both cases valgrind does not detect any memory leak. The amount of still reachable memory differs by about 8x10^7 bytes, i.e. the size of the original vector in l$a. This demonstrates that R indeed knows about the original vector and frees it when it is told to do so, but this would also happen when R decides by itself to run the garbage collector.
I have a program like this:
int *number0 = new int;
int main()
{
int *number1 = new int;
}
I thought, both memory allocations would introduce memory leaks, though valgrind only
complains about number1 inside the main function. Why is that?
Running this
int *x = new int;
int main()
{
return 0;
}
code (i.e without the leak in the main, compiled with g++ 4.8.1) using valgrind (3.8.1) with (-v --track-origins=yes --leak-check=full --show-reachable=yes) I get:
==34301==
==34301== HEAP SUMMARY:
==34301== in use at exit: 4 bytes in 1 blocks
==34301== total heap usage: 1 allocs, 0 frees, 4 bytes allocated
==34301==
==34301== Searching for pointers to 1 not-freed blocks
==34301== Checked 189,064 bytes
==34301==
==34301== 4 bytes in 1 blocks are still reachable in loss record 1 of 1
==34301== at 0x4C2A879: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==34301==
==34301== LEAK SUMMARY:
==34301== definitely lost: 0 bytes in 0 blocks
==34301== indirectly lost: 0 bytes in 0 blocks
==34301== possibly lost: 0 bytes in 0 blocks
==34301== still reachable: 4 bytes in 1 blocks
==34301== suppressed: 0 bytes in 0 blocks
Which means that you should also pay attention to the in use at exit category.
It doesn't seem like valgrind missed it, just placed it in another category, possibly because they assume something as lost only when you cannot by any means track down that address and release it, but this variable is never lost.
This however :
int *x = new int;
int main()
{
x = new int;
return 0;
}
Is detected as a leak as you really lose track of the memory you've allocated.
EDIT: As described in the Mem-check manual:
"Still reachable". This covers cases 1 and 2 (for the BBB blocks)
above. A start-pointer or chain of start-pointers to the block is
found. Since the block is still pointed at, the programmer could, at
least in principle, have freed it before program exit. "Still
reachable" blocks are very common and arguably not a problem. So, by
default, Memcheck won't report such blocks individually.
So as noted before they do detect it, they just think it's less exciting
Valgrind is not a perfect tool for detecting the presence or absence of all possible memory leaks, but rather a helpful tool that can detect some memory leaks. This means that valgrind output cannot be used to determine whether a particular piece of code contains or does not contain any leaks.
Neither of your news have corresponding deletes, and in that sense, they are both leaked.
It is likely that valgrind considers the number0 memory to not be leaked because the memory it points to is reachable at the end of the program execution. In contrast to this, number1 goes out of scope, and so the memory it points to is not reachable at the end of the execution of your program, and so valgrind considers it to be leaked.
I see two leaks using valgrind 3.8.1 / g++ 4.8.1:
$ g++ -o foo -g foo.cc
$ valgrind ./foo
...
==7789== HEAP SUMMARY:
==7789== in use at exit: 8 bytes in 2 blocks
==7789== total heap usage: 2 allocs, 0 frees, 8 bytes allocated
==7789==
==7789== LEAK SUMMARY:
==7789== definitely lost: 4 bytes in 1 blocks
==7789== indirectly lost: 0 bytes in 0 blocks
==7789== possibly lost: 0 bytes in 0 blocks
==7789== still reachable: 4 bytes in 1 blocks
==7789== suppressed: 0 bytes in 0 blocks
The "definitely lost" bytes are in main in my test.
This
int *number0 = new int;
is not a memory leak because it gets reclaimed at the end of execution.
This configuration is a [potential] memory leak because
int main()
{
int *number1 = new int;
}
some other part of the code could call
main () ;
and it could be called repeatedly.
I wrote the following code to get a basic understanding of Valgrind and having a hard time interpreting its output. This probably is not related to Valgrind but more basic C++.
#include <string>
#include <iostream>
using namespace std;
class Valgrind_testclass
{
std::string * stringInHeap;
public:
Valgrind_testclass() {
stringInHeap = new std::string("String in heap");
}
~Valgrind_testclass() {
//delete stringInHeap;
}
void PrintFunc(void) {
cout << "Nothing but a printout" << endl;
}
};
int main()
{
Valgrind_testclass * valObjPtr = new Valgrind_testclass();
delete valObjPtr;
return 0;
}
Valgrind outputs:
==4459== HEAP SUMMARY:
==4459== in use at exit: 31 bytes in 2 blocks
==4459== total heap usage: 3 allocs, 1 frees, 35 bytes allocated
==4459==
==4459== Searching for pointers to 2 not-freed blocks
==4459== Checked 102,100 bytes
==4459==
==4459== 31 (4 direct, 27 indirect) bytes in 1 blocks are definitely lost in loss record 2 of 2
==4459== at 0x402641D: operator new(unsigned int) (vg_replace_malloc.c:255)
==4459== by 0x80487DB: Valgrind_testclass::Valgrind_testclass() (in /home/madu/C++/ValgrindTest)
==4459== by 0x80486F6: main (in /home/madu/C++/ValgrindTest)
==4459==
==4459== LEAK SUMMARY:
==4459== definitely lost: 4 bytes in 1 blocks
==4459== indirectly lost: 27 bytes in 1 blocks
==4459== possibly lost: 0 bytes in 0 blocks
==4459== still reachable: 0 bytes in 0 blocks
==4459== suppressed: 0 bytes in 0 blocks
==4459==
==4459== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 17 from 6)
Could someone please tell me where I do 3 allocations? I can only see two allocations. Also why it says "indirectly lost"?
Thank you.
When you construct an std::string object, it allocates another pointer (internal to the object) to point to the string value. This is where the third allocation is coming from, and is also the indirectly-leaked memory.
In other words, you have these three allocations:
new Valgrind_testclass() (explicit)
new std::string("String in heap") (explicit)
The string-internal allocation (implicit / indirect)
Since you leaked allocation 2, you indirectly leaked allocation 3 as well; the string's destructor will not be called, and therefore it has no opportunity to free allocation 3.
You have 3 allocations because std::string also allocates memory.
Indirectly lost, means that you lost a pointer to something that had a pointer to some other memory. In this case, you didn't delete stringInHeap, and you lost its pointer. Through that, the actual std::string who had allocated memory could not delete it and therefore that memory is also lost.