I am trying to find a memory leak in my code but for the life of me, I cannot figure out or identify the reason for this leak. Which is why I am reaching out to your all. Thanks in advance.
The following code, creates a kerberos cache from keytab. The code is an extract of a bigger code, but this also has the same memory leak. The full code is as following
#include <iostream>
#include <cstring>
#include <krb5/krb5.h>
bool renew(krb5_context &_ctx, krb5_keytab &_keytab, krb5_ccache &_cache, std::string &_principal)
{
long int retval;
bool success = false;
krb5_principal principal;
krb5_creds *creds;
if ((retval = krb5_parse_name(_ctx, _principal.c_str(), &principal)))
throw "cannot parse principal string";
creds = (krb5_creds*) malloc(sizeof(*creds));
memset(creds, 0, sizeof(creds));
if ((retval = krb5_get_init_creds_keytab(_ctx, creds, principal, _keytab, 0, NULL, NULL)))
{
free(creds);
krb5_free_principal(_ctx, principal);
throw "cannot initialize keytab credentials - ";
}
if ((retval = krb5_cc_initialize(_ctx, _cache, principal)))
{
free(creds);
krb5_free_principal(_ctx, principal);
throw " cannot initialize cache - ";
}
if ((retval = krb5_cc_store_cred(_ctx, _cache, creds)))
{
free(creds);
krb5_free_principal(_ctx, principal);
throw "cannot store credentials - ";
}
free(creds);
krb5_free_principal(_ctx, principal);
return success;
}
int main()
{
long int retval;
std::string _keytab_file, _cache_file, _realm, _principal;
krb5_context _ctx = NULL;
krb5_keytab _keytab = NULL;
krb5_ccache _cache = NULL;
_keytab_file = "/location/to/the/keytab/file";
_cache_file = "/location/to/the/cache/file";
_principal = "user.name#DOMAIN.COM";
if ((retval = krb5_init_context(&_ctx)))
throw "cannot initialize context";
if ((retval = krb5_kt_resolve(_ctx, _keytab_file.c_str(), &_keytab)))
throw "cannot resolve keytab";
if ((retval = krb5_cc_resolve(_ctx, _cache_file.c_str(), &_cache)))
throw "cannot open/initialize kerberos cache";
try {
renew(_ctx, _keytab, _cache, _principal);
} catch (std::exception &e) {
std::cerr<<e.what()<<std::endl;
}
bailout:
if (_cache)
krb5_cc_close(_ctx, _cache);
if (_keytab)
krb5_kt_close(_ctx, _keytab);
if (_ctx)
krb5_free_context(_ctx);
return 0;
}
in a linux (or similar) box following used to compile and run with valgrind to check memory leak
g++ krb.cpp -o krb -lkrb5 -g
valgrind --leak-check=full ./krb
the valgrind report is as following-
==257623==
==257623== HEAP SUMMARY:
==257623== in use at exit: 4,964 bytes in 19 blocks
==257623== total heap usage: 9,165 allocs, 9,146 frees, 3,702,330 bytes allocated
==257623==
==257623== 16 bytes in 1 blocks are definitely lost in loss record 6 of 14
==257623== at 0x4C37135: malloc (vg_replace_malloc.c:381)
==257623== by 0x5E31F8D: krb5int_c_copy_keyblock_contents (in /usr/lib64/libk5crypto.so.3.1)
==257623== by 0x4E91146: ??? (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E9C65B: ??? (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E9C6F5: krb5_get_init_creds_keytab (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4012EF: renew(_krb5_context*&, _krb5_kt*&, _krb5_ccache*&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (krb.cpp:25)
==257623== by 0x401612: main (krb.cpp:76)
==257623==
==257623== 78 (40 direct, 38 indirect) bytes in 1 blocks are definitely lost in loss record 8 of 14
==257623== at 0x4C37135: malloc (vg_replace_malloc.c:381)
==257623== by 0x4E91574: krb5_copy_principal (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E910EB: ??? (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E9C65B: ??? (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E9C6F5: krb5_get_init_creds_keytab (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4012EF: renew(_krb5_context*&, _krb5_kt*&, _krb5_ccache*&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (krb.cpp:25)
==257623== by 0x401612: main (krb.cpp:76)
==257623==
==257623== 97 (40 direct, 57 indirect) bytes in 1 blocks are definitely lost in loss record 9 of 14
==257623== at 0x4C37135: malloc (vg_replace_malloc.c:381)
==257623== by 0x4E91574: krb5_copy_principal (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E9112F: ??? (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E9C65B: ??? (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E9C6F5: krb5_get_init_creds_keytab (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4012EF: renew(_krb5_context*&, _krb5_kt*&, _krb5_ccache*&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (krb.cpp:25)
==257623== by 0x401612: main (krb.cpp:76)
==257623==
==257623== 1,646 bytes in 1 blocks are definitely lost in loss record 13 of 14
==257623== at 0x4C37135: malloc (vg_replace_malloc.c:381)
==257623== by 0x4E91369: krb5int_copy_data_contents (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E913EB: krb5_copy_data (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E91198: ??? (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E9C65B: ??? (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4E9C6F5: krb5_get_init_creds_keytab (in /usr/lib64/libkrb5.so.3.3)
==257623== by 0x4012EF: renew(_krb5_context*&, _krb5_kt*&, _krb5_ccache*&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (krb.cpp:25)
==257623== by 0x401612: main (krb.cpp:76)
==257623==
==257623== LEAK SUMMARY:
==257623== definitely lost: 1,742 bytes in 4 blocks
==257623== indirectly lost: 95 bytes in 7 blocks
==257623== possibly lost: 0 bytes in 0 blocks
==257623== still reachable: 3,127 bytes in 8 blocks
==257623== suppressed: 0 bytes in 0 blocks
==257623== Reachable blocks (those to which a pointer was found) are not shown.
==257623== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==257623==
==257623== For lists of detected and suppressed errors, rerun with: -s
==257623== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)
looking forward to your input :)
I have a thread system that keeps allways opened a certain amount of threads. The problem with this program si that it is having some memory leak problems. I am starting the program at around 8MB (RAM memory) and in 1 day i see it at 800MB. I used valgrind to track down this issue and i found the following while running this command time valgrind --leak-check=yes ./test2; :
I get this when i use 1000 treads and it is not even finishing.
Thread 157: status = VgTs_WaitSys
==16172== at 0x4160F8B: ??? (in /lib/libc-2.5.so)
==16172== by 0x40FF58F: _IO_file_underflow##GLIBC_2.1 (in /lib/libc-2.5.so)
==16172== by 0x40FFC8A: _IO_default_uflow (in /lib/libc-2.5.so)
==16172== by 0x410100C: __uflow (in /lib/libc-2.5.so)
==16172== by 0x40F48B5: _IO_getline_info (in /lib/libc-2.5.so)
==16172== by 0x40F4800: _IO_getline (in /lib/libc-2.5.so)
==16172== by 0x40F3779: fgets (in /lib/libc-2.5.so)
==16172== by 0x80496CF: exec(char const*) (in /home/pthread_test)
==16172== by 0x80498C2: thread_test(void*) (in /home/pthread_test)
==16172== by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==16172== by 0x417047D: clone (in /lib/libc-2.5.so)
Thread 158: status = VgTs_WaitSys
==16172== at 0x4160F8B: ??? (in /lib/libc-2.5.so)
==16172== by 0x40FF58F: _IO_file_underflow##GLIBC_2.1 (in /lib/libc-2.5.so)
==16172== by 0x40FFC8A: _IO_default_uflow (in /lib/libc-2.5.so)
==16172== by 0x410100C: __uflow (in /lib/libc-2.5.so)
==16172== by 0x40F48B5: _IO_getline_info (in /lib/libc-2.5.so)
==16172== by 0x40F4800: _IO_getline (in /lib/libc-2.5.so)
==16172== by 0x40F3779: fgets (in /lib/libc-2.5.so)
==16172== by 0x80496CF: exec(char const*) (in /home/pthread_test)
==16172== by 0x80498C2: thread_test(void*) (in /home/pthread_test)
==16172== by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==16172== by 0x417047D: clone (in /lib/libc-2.5.so)
When using just ./test2 (running without valgrind), absolutely all threads finishes and i can see in the debug.log that every thread finishes
tail debug.log
END:1972:191
END:1984:191
END:1992:191
END:1974:191
END:1986:191
END:1979:191
END:1993:191
END:1983:191
END:1999:191 <-- even thread 1999 finished
END:1991:191
When i am doing with 250 threads i get this at the end of the program
THREAD [1993] ending
THREAD [1983] ending
THREAD [1999] ending
THREAD [1991] ending
==23152== Thread 25:
==23152== Invalid free() / delete / delete[]
==23152== at 0x400579D: free (vg_replace_malloc.c:325)
==23152== by 0x3B59DD: ??? (in /lib/libc-2.5.so)
==23152== by 0x3B5556: ??? (in /lib/libc-2.5.so)
==23152== by 0x40013B2: _vgnU_freeres (vg_preloaded.c:62)
==23152== by 0x333A73: _Exit (in /lib/libc-2.5.so)
==23152== by 0x40AAF8: start_thread (in /lib/libpthread-2.5.so)
==23152== by 0x37547D: clone (in /lib/libc-2.5.so)
==23152== Address 0x409b2c8 is not stack'd, malloc'd or (recently) free'd
==23152==
==23152==
==23152== HEAP SUMMARY:
==23152== in use at exit: 160,850 bytes in 12,002 blocks
==23152== total heap usage: 20,151 allocs, 8,150 frees, 2,640,139 bytes allocated
==23152==
==23152== Thread 1:
==23152== 152 bytes in 1 blocks are possibly lost in loss record 2 of 8
==23152== at 0x4004EC2: calloc (vg_replace_malloc.c:418)
==23152== by 0x291FF9: _dl_allocate_tls (in /lib/ld-2.5.so)
==23152== by 0x40B026: pthread_create##GLIBC_2.1 (in /lib/libpthread-2.5.so)
==23152== by 0x80493DF: main (in /home/freelancer1/pthreads_test/test2)
==23152==
==23152== 20,000 bytes in 2,000 blocks are definitely lost in loss record 3 of 8
==23152== at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152== by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152== by 0x8049C56: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152== by 0x804981D: thread_test(void*) (in /home/freelancer1/pthreads_test/test2)
==23152== by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==23152== by 0x37547D: clone (in /lib/libc-2.5.so)
==23152==
==23152== 20,000 bytes in 2,000 blocks are definitely lost in loss record 4 of 8
==23152== at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152== by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152== by 0x8049C56: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152== by 0x804944D: main (in /home/freelancer1/pthreads_test/test2)
==23152==
==23152== 20,000 bytes in 2,000 blocks are definitely lost in loss record 5 of 8
==23152== at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152== by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152== by 0x8049C56: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152== by 0x8049950: thread_test(void*) (in /home/freelancer1/pthreads_test/test2)
==23152== by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==23152== by 0x37547D: clone (in /lib/libc-2.5.so)
==23152==
==23152== 26,890 bytes in 2,000 blocks are definitely lost in loss record 6 of 8
==23152== at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152== by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152== by 0x8049C64: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152== by 0x8049950: thread_test(void*) (in /home/freelancer1/pthreads_test/test2)
==23152== by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==23152== by 0x37547D: clone (in /lib/libc-2.5.so)
==23152==
==23152== 30,890 bytes in 2,000 blocks are definitely lost in loss record 7 of 8
==23152== at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152== by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152== by 0x8049C64: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152== by 0x804981D: thread_test(void*) (in /home/freelancer1/pthreads_test/test2)
==23152== by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==23152== by 0x37547D: clone (in /lib/libc-2.5.so)
==23152==
==23152== 42,890 bytes in 2,000 blocks are definitely lost in loss record 8 of 8
==23152== at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152== by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152== by 0x8049C64: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152== by 0x804944D: main (in /home/freelancer1/pthreads_test/test2)
==23152==
==23152== LEAK SUMMARY:
==23152== definitely lost: 160,670 bytes in 12,000 blocks
==23152== indirectly lost: 0 bytes in 0 blocks
==23152== possibly lost: 152 bytes in 1 blocks
==23152== still reachable: 28 bytes in 1 blocks
==23152== suppressed: 0 bytes in 0 blocks
==23152== Reachable blocks (those to which a pointer was found) are not shown.
==23152== To see them, rerun with: --leak-check=full --show-reachable=yes
==23152==
==23152== For counts of detected and suppressed errors, rerun with: -v
==23152== ERROR SUMMARY: 8 errors from 8 contexts (suppressed: 36 from 8)
Here is the source code :
#include <iostream>
#include <cstdlib>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <cstring>
#include <stdexcept>
using namespace std;
void file_put_contents(char* filename, char* content, char* mode)
{
char* filename2;
filename2 = strdup(filename);
char* content2;
content2 = strdup(content);
char* writemode = "a";
if(strstr(mode,"FILE_APPEND"))
{
writemode = "a";
}
else if(strstr(mode,"FILE_OVERWRITE"))
{
writemode = "w";
}
else
{
writemode = "w";
}
FILE * file;
file = fopen(filename2,writemode);
fprintf(file,content2);
fclose(file);
}
int NUM_THREADS = 2000;
int MAX_THREADS = 500;
int THREADSTACK = 65536;
struct thread_struct{
int arg1;
int arg2;
};
pthread_mutex_t mutex_;
pthread_mutex_t mutex2_;
static unsigned int thread_count = 0;
string exec(const char* cmd)
{
int DEBUG=0;
char buffer[5000];
string result = "";
FILE* pipe = popen(cmd, "r");
if (!pipe && DEBUG) throw runtime_error("popen() failed!");
try
{
while (!feof(pipe))
{
if (fgets(buffer, 128, pipe) != NULL)
{
result += buffer;
}
}
}
catch(...)
{
pclose(pipe);
throw;
}
pclose(pipe);
return result;
}
void *thread_test(void *arguments)
{
pthread_mutex_lock(&mutex_);
thread_count++;
pthread_mutex_unlock(&mutex_);
struct thread_struct *args = (thread_struct*)arguments;
int thread_id = (int) args->arg1;
char debug_var[100];
// UGLY DEBUG
memset(debug_var,0,sizeof(debug_var)); sprintf(debug_var,"BEGIN:%d:%d\n",thread_id,__LINE__); file_put_contents("debug.log",debug_var,"FILE_APPEND");
int random_sleep;
random_sleep = rand() % 10 + 3;
char command[100];
memset(command,0,sizeof(command));
sprintf(command,"sleep %d",random_sleep);
exec(command);
pthread_mutex_lock(&mutex_);
thread_count--;
pthread_mutex_unlock(&mutex_);
// UGLY DEBUG
memset(debug_var,0,sizeof(debug_var)); sprintf(debug_var,"END:%d:%d\n",thread_id,__LINE__); file_put_contents("debug.log",debug_var,"FILE_APPEND");
printf("THREAD [%d] ending\n",thread_id);
pthread_exit(NULL);
}
int main()
{
// pthread_t threads[NUM_THREADS];
int rc;
printf("Beginning ...");
usleep(1000000);
srand ((unsigned)time(NULL));
unsigned int thread_count_now = 0;
pthread_attr_t attrs;
pthread_attr_init(&attrs);
pthread_attr_setstacksize(&attrs, THREADSTACK);
pthread_attr_setdetachstate(&attrs, PTHREAD_CREATE_DETACHED);
pthread_mutex_init(&mutex2_, NULL);
for(int i=0; i < NUM_THREADS; i++ )
{
create_thread:
pthread_mutex_lock(&mutex_);
thread_count_now = thread_count;
pthread_mutex_unlock(&mutex_);
struct thread_struct struct1;
struct1.arg1 = i;
struct1.arg2 = 999;
pthread_t temp_thread;
printf("CREATE thread [%d]\n",i);
rc = pthread_create(&temp_thread, &attrs, &thread_test, (void *)&struct1);
char debug_var[100];
memset(debug_var,0,sizeof(debug_var)); sprintf(debug_var,"THREAD:%d:RC=%d:%d\n",i,rc,__LINE__); file_put_contents("debug.log",debug_var,"FILE_APPEND");
if (rc)
{
printf("Unable to create thread %d\n",rc);
sleep(1);
pthread_detach(temp_thread);
goto create_thread;
}
usleep(10);
while(true)
{
if(thread_count >= MAX_THREADS)
{
// printf("Thread POOL full %d of %d\n",thread_count,MAX_THREADS);
}
else
{
break;
}
usleep(10);
}
}
pthread_attr_destroy(&attrs);
pthread_mutex_destroy(&mutex2_);
printf("Proccess completed!\n");
pthread_exit(NULL);
return 1;
}
Did i forgot somewhere to free memory?
I don't quite understand those logs perfectly, can you give me any clue? What is wrong? What do i have to change?
Thank you.
I have implemented multitask application in c++. Producer push on queue, and consumer get elements from queue. Sometimes my application crashed. Could someone help me with this problem. sf
Valgrind output:
==10769== Memcheck, a memory error detector
==10769== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==10769== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==10769== Command: ./tachyon -s /HOME_ann/BII/biidurgak/test_new_tachyon/Tachyon_v5_improvedQueryTime/settings_smallNR
==10769==
==10769== Thread 5:
==10769== Invalid read of size 4
==10769== at 0x342669C9DE: std::string::assign(std::string const&) (in /usr/lib64/libstdc++.so.6.0.8)
==10769== by 0x42E01A: std::pair::operator=(std::pair const&) (stl_pair.h:152)
==10769== by 0x42B27A: boost::lockfree::detail::ringbuffer_base >::pop(std::pair&, std::pair*, unsigned
long) (spsc_queue.hpp:154)
==10769== by 0x428719: boost::lockfree::detail::compile_time_sized_ringbuffer, 2ul>::pop(std::pair&) (spsc_queue.hpp:305)
==10769== by 0x425DEA: boost::lockfree::spsc_queue, boost::lockfree::capacity, boost::parameter::void_>::pop(std::pair&) (spsc_queue.hpp:572)
==10769== by 0x41BE16: findInDatabase() (Tachyon.cpp:103)
==10769== by 0x4351D4: boost::detail::thread_data::run() (thread.hpp:117)
==10769== by 0x4E53D01: thread_proxy (in /HOME_ann/BII/biidurgak/test_new_tachyon/boost_install/boost_1_55_0/lib/libboost_thread.so.1.55.0)
==10769== by 0x342120673C: start_thread (in /lib64/libpthread-2.5.so)
==10769== by 0x34206D3D1C: clone (in /lib64/libc-2.5.so)
==10769== Address 0xfffffffffffffff8 is not stack'd, malloc'd or (recently) free'd
==10769==
==10769==
==10769== Process terminating with default action of signal 11 (SIGSEGV)
==10769== Access not within mapped region at address 0xFFFFFFFFFFFFFFF8
==10769== at 0x342669C9DE: std::string::assign(std::string const&) (in /usr/lib64/libstdc++.so.6.0.8)
==10769== by 0x42E01A: std::pair::operator=(std::pair const&) (stl_pair.h:152)
==10769== by 0x42B27A: boost::lockfree::detail::ringbuffer_base >::pop(std::pair&, std::pair*, unsigned long) (spsc_queue.hpp:154)
==10769== by 0x428719: boost::lockfree::detail::compile_time_sized_ringbuffer, 2ul>::pop(std::pair&) (spsc_queue.hpp:305)
==10769== by 0x425DEA: boost::lockfree::spsc_queue, boost::lockfree::capacity, boost::parameter::void_>::pop(std::pair&) (spsc_queue.hpp:572)
==10769== by 0x41BE16: findInDatabase() (Tachyon.cpp:103)
==10769== by 0x4351D4: boost::detail::thread_data::run() (thread.hpp:117)
==10769== by 0x4E53D01: thread_proxy (in /HOME_ann/BII/biidurgak/test_new_tachyon/boost_install/boost_1_55_0/lib/libboost_thread.so.1.55.0)
==10769== by 0x342120673C: start_thread (in /lib64/libpthread-2.5.so)
==10769== by 0x34206D3D1C: clone (in /lib64/libc-2.5.so)
==10769== If you believe this happened as a result of a stack
==10769== overflow in your program's main thread (unlikely but
==10769== possible), you can try to increase the size of the
==10769== main thread stack using the --main-stacksize= flag.
==10769== The main thread stack size used in this run was 10485760.
==10769==
==10769== HEAP SUMMARY:
==10769== in use at exit: 219,895,341 bytes in 4,598,508 blocks
==10769== total heap usage: 36,680,650 allocs, 32,082,142 frees, 1,474,244,383 bytes allocated
==10769==
==10769== LEAK SUMMARY:
==10769== definitely lost: 1,904 bytes in 2 blocks
==10769== indirectly lost: 0 bytes in 0 blocks
==10769== possibly lost: 184,232,229 bytes in 4,598,462 blocks
==10769== still reachable: 35,661,208 bytes in 44 blocks
==10769== suppressed: 0 bytes in 0 blocks
==10769== Rerun with --leak-check=full to see details of leaked memory
==10769==
==10769== For counts of detected and suppressed errors, rerun with: -v
Producer:
void producer(string file) {
ifstream query(file.c_str());
string description = "";
string sequence = "";
string line;
while (getline(query, line)) {
//read description
if (line == "") continue;
if (line.at(0) == '>') {
if (sequence != "") {
pair<string, string> a = make_pair(description, sequence);
while (!queue.push(a))
;
sequence = "";
}
description = line.substr(1);
} else {
sequence += line;
}
}
if (sequence != "" && description != "") {
pair<string, string> a = make_pair(description, sequence);
while (!queue.push(a))
;
}
}
In the consumer I have this:
void Consumer(void) {
pair<string, string>element;
//part of code
while(queue.pop(element)){ //Line 103 in Tachyon.cpp
string queryDescription = element.first;
string sequence = element.second;
//Part of code
}
}
Queue is the global variable:
boost::lockfree::spsc_queue<pair<string, string>, boost::lockfree::capacity<2> > queue;
Global variables, multiple threads (I assume), one reading and another thread writing - you need a synchronization object like mutex or critical section.
Pseudocode:
// Consumer
loop-begin;
Lock();
get-item-into-local-variable
Unlock();
process-local-variable-item;
loop-end
// Producer
void AddItem(item)
{
Lock();
Add-item-into-queue
Unlock();
}
I've been trying to debug this for a month, sure it was my bad programming practise, but I think it may be a bug, so I'm asking here first before I report.
Consider the following code:
#include <sys/resource.h> // memory management.
#include <stdio.h>
#include <iostream>
#include <iomanip>
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/core/core.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/video/background_segm.hpp"
using namespace std;
using namespace cv;
// Load frame from disk.
void readFrame(int frameNum, Mat &frame) {
// Construct filenames
Mat image;
stringstream number, filename;
number << setw(7) << setfill('0') << frameNum; // expecting over 1e10 images over the installation period.
filename << "../images/store-" << number.str() << ".jpg"; // assumes jpegs!//
cout << "Loading filename: " << filename.str() << endl;
image = imread( filename.str() );
if (image.empty() or !image.data) {
cout << "Input image empty:\n";
}
frame = image.clone();
}
// Class to hold the perceptual chunks.
class percepUnit {
public:
cv::Mat image; // percept itself
cv::Mat mask; // alpha channel
// constructor method
percepUnit(cv::Mat &ROI, cv::Mat &alpha, int ix, int iy, int iw, int ih, int area) {
image = ROI.clone();
mask = alpha.clone();
}
};
// Segment foreground from background
void segmentForeground(list<percepUnit*> &percepUnitsForeground, Mat &foreground, Mat &frame) {
Mat contourImage = Mat(foreground.rows, foreground.cols, CV_8UC1, Scalar::all(0));
vector<vector<Point>> contours;
int area;
// The following causes strange spikes in memory usage:
// find contours
findContours(foreground, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
for (int idx = 0; idx < contours.size(); idx++) {
area = contourArea(contours[idx]);
if (area > 100) {
percepUnit *thisUnit = new percepUnit(frame, contourImage, 0, 0, 100,100, area);
percepUnitsForeground.push_back(thisUnit); // Append to percepUnits
}
}
/* The following does not:
findContours(foreground, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
for (int idx = 0; idx < contours.size(); idx++) {
area = contourArea(contours[idx]);
}*/
/* Neither does this:
for (int idx = 0; idx < 10; idx++) {
percepUnit *thisUnit = new percepUnit(frame, contourImage, 0, 0, 100,100, area);
percepUnitsForeground.push_back(thisUnit); // Append to percepUnits
}*/
}
int main(int argc, const char** argv)
{
int frameCount = 78298;
Mat frame, foreground;
BackgroundSubtractorMOG2 MOG2model;
list<percepUnit*> scratchPercepUnitsForeground;
// add rusage stuff
struct rusage usage; // memory usage.
for(int i=0; i<= 75; i++)
{
// run full segmenter here. (background disabled)
readFrame(frameCount, frame); // was frame = readFrame();
// Only process if this frame actually loaded (non empty)
if ( not frame.empty() ) {
MOG2model(frame,foreground); // Update MOG2 model, downscale?
// before we segment again clear scratch
// TODO how to delete the actual memory allocated? Run delete on everything?
for (list<percepUnit*>::iterator percepIter = scratchPercepUnitsForeground.begin();
percepIter != scratchPercepUnitsForeground.end();
percepIter++) {
delete *percepIter; // delete what we point to.
//percepIter = scratchPercepUnitsForeground.erase(percepIter); // remove the pointer itself, and update the iterator.
}
// Added with EDIT1
scratchPercepUnitsForeground.clear();
// Segment the foreground regions and generate boolImage to extract from background.
segmentForeground(scratchPercepUnitsForeground, foreground, frame);
}
frameCount++;
getrusage(RUSAGE_SELF, &usage);
cout << "DEBUG leakTest_bug_report " << i << " " << usage.ru_maxrss/1024.0 << endl;
}
return 0;
}
If you use the images available here (http://www.ekran.org/tmp/images.tar.gz), you will find that the program's memory usage increases, and it seems to increase with the number of foreground contours. Since I'm clearing my storage (scratchPercepUnitsForeground) for each frame, I don't see why there should be increasing memory usage. The segmentForeground() function should exit, deallocating all its used memory, for each frame. The memory usage should be constant over time since we only check memory usage after the function has exited. It seems something is being left over that I can't figure out.
If I run just the findContours() part without the percepUnit() constructor, memory usage is constant, as I expect. If I run just the percepUnit() constructor without findContours(), memory usage is constant. Memory usage increases only when I use both. See commented code in segmentForeground() above.
I've confirmed this issue on two of my machines (both AMD64, linux) and running opencv 2.4.6.1 and 2.4.5.
EDIT1
Code above has been changed to include the suggestion below, and yet the problem persists.
Here is what the memory increase looks like:
(source: ekran.org)
The red line is the increase of memory (which is correlated with the test images linked above) seen when findContours() and the constructor are both called. The stable lines below are the two cases where we run either findContours() or the constructor.
Valgrind Output
==2055== Memcheck, a memory error detector
==2055== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==2055== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==2055== Command: ./leakTest
==2055==
==2055==
==2055== HEAP SUMMARY:
==2055== in use at exit: 217,751,704 bytes in 112 blocks
==2055== total heap usage: 800,066 allocs, 799,954 frees, 29,269,767,865 bytes allocated
==2055==
==2055== 568 bytes in 1 blocks are still reachable in loss record 1 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x63A720A: __fopen_internal (iofopen.c:76)
==2055== by 0xA8BC050: libjpeg_general_init (in /usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2)
==2055== by 0x400F305: call_init.part.0 (dl-init.c:85)
==2055== by 0x400F3DE: _dl_init (dl-init.c:52)
==2055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==2055==
==2055== 2,072 bytes in 1 blocks are still reachable in loss record 2 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x1495F675: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x1495E4AE: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x14950888: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x149253B8: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x400F305: call_init.part.0 (dl-init.c:85)
==2055== by 0x400F3DE: _dl_init (dl-init.c:52)
==2055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==2055==
==2055== 2,072 bytes in 1 blocks are still reachable in loss record 3 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x1495F675: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x1495E0EF: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x14950890: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x149253B8: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x400F305: call_init.part.0 (dl-init.c:85)
==2055== by 0x400F3DE: _dl_init (dl-init.c:52)
==2055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==2055==
==2055== 2,072 bytes in 1 blocks are still reachable in loss record 4 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x1495F675: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x14971A6F: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x14950898: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x149253B8: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x400F305: call_init.part.0 (dl-init.c:85)
==2055== by 0x400F3DE: _dl_init (dl-init.c:52)
==2055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==2055==
==2055== 2,072 bytes in 1 blocks are still reachable in loss record 5 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x1495F675: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x1499024F: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x149508A0: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x149253B8: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x400F305: call_init.part.0 (dl-init.c:85)
==2055== by 0x400F3DE: _dl_init (dl-init.c:52)
==2055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==2055==
==2055== 2,072 bytes in 1 blocks are still reachable in loss record 6 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x1495F675: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x149610EF: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x149253B8: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==2055== by 0x400F305: call_init.part.0 (dl-init.c:85)
==2055== by 0x400F3DE: _dl_init (dl-init.c:52)
==2055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==2055==
==2055== 4,096 bytes in 1 blocks are still reachable in loss record 7 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0xA8BC067: libjpeg_general_init (in /usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2)
==2055== by 0x400F305: call_init.part.0 (dl-init.c:85)
==2055== by 0x400F3DE: _dl_init (dl-init.c:52)
==2055== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==2055==
==2055== 1,555,228 bytes in 1 blocks are possibly lost in loss record 8 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x4E87A90: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4ECDBF1: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4ECE378: cv::_OutputArray::create(int, int, int, int, bool, int) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4F52F7D: cv::Mat::copyTo(cv::_OutputArray const&) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x40253C: cv::Mat::clone() const (mat.hpp:335)
==2055== by 0x4028D5: percepUnit::percepUnit(cv::Mat&, cv::Mat&, int, int, int, int, int) (leakTest.cpp:43)
==2055== by 0x401E0E: segmentForeground(std::list<percepUnit*, std::allocator<percepUnit*> >&, cv::Mat&, cv::Mat&) (leakTest.cpp:63)
==2055== by 0x40202D: main (leakTest.cpp:114)
==2055==
==2055== 37,325,024 bytes in 8 blocks are possibly lost in loss record 9 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x4E87A90: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4ECDBF1: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4ECE378: cv::_OutputArray::create(int, int, int, int, bool, int) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4F52F7D: cv::Mat::copyTo(cv::_OutputArray const&) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x40253C: cv::Mat::clone() const (mat.hpp:335)
==2055== by 0x402897: percepUnit::percepUnit(cv::Mat&, cv::Mat&, int, int, int, int, int) (leakTest.cpp:42)
==2055== by 0x401E0E: segmentForeground(std::list<percepUnit*, std::allocator<percepUnit*> >&, cv::Mat&, cv::Mat&) (leakTest.cpp:63)
==2055== by 0x40202D: main (leakTest.cpp:114)
==2055==
==2055== 52,877,752 bytes in 34 blocks are indirectly lost in loss record 10 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x4E87A90: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4ECDBF1: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4ECE378: cv::_OutputArray::create(int, int, int, int, bool, int) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4F52F7D: cv::Mat::copyTo(cv::_OutputArray const&) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x40253C: cv::Mat::clone() const (mat.hpp:335)
==2055== by 0x4028D5: percepUnit::percepUnit(cv::Mat&, cv::Mat&, int, int, int, int, int) (leakTest.cpp:43)
==2055== by 0x401E0E: segmentForeground(std::list<percepUnit*, std::allocator<percepUnit*> >&, cv::Mat&, cv::Mat&) (leakTest.cpp:63)
==2055== by 0x40202D: main (leakTest.cpp:114)
==2055==
==2055== 125,971,956 bytes in 27 blocks are indirectly lost in loss record 11 of 12
==2055== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x4E87A90: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4ECDBF1: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4ECE378: cv::_OutputArray::create(int, int, int, int, bool, int) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x4F52F7D: cv::Mat::copyTo(cv::_OutputArray const&) const (in /usr/local/lib/libopencv_core.so.2.4.5)
==2055== by 0x40253C: cv::Mat::clone() const (mat.hpp:335)
==2055== by 0x402897: percepUnit::percepUnit(cv::Mat&, cv::Mat&, int, int, int, int, int) (leakTest.cpp:42)
==2055== by 0x401E0E: segmentForeground(std::list<percepUnit*, std::allocator<percepUnit*> >&, cv::Mat&, cv::Mat&) (leakTest.cpp:63)
==2055== by 0x40202D: main (leakTest.cpp:114)
==2055==
==2055== 178,856,428 (6,720 direct, 178,849,708 indirect) bytes in 35 blocks are definitely lost in loss record 12 of 12
==2055== at 0x4C2B1C7: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2055== by 0x401DD3: segmentForeground(std::list<percepUnit*, std::allocator<percepUnit*> >&, cv::Mat&, cv::Mat&) (leakTest.cpp:63)
==2055== by 0x40202D: main (leakTest.cpp:114)
==2055==
==2055== LEAK SUMMARY:
==2055== definitely lost: 6,720 bytes in 35 blocks
==2055== indirectly lost: 178,849,708 bytes in 61 blocks
==2055== possibly lost: 38,880,252 bytes in 9 blocks
==2055== still reachable: 15,024 bytes in 7 blocks
==2055== suppressed: 0 bytes in 0 blocks
==2055==
==2055== For counts of detected and suppressed errors, rerun with: -v
==2055== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 2 from 2)
So basically this seems to tell us that there could be a problem in the percepUnit constructor that does clone(). Running the constructor without findContours() shows no memory increase (as stated above), which includes the use of "new". The jpeg reader has also been unit tested, no memory increase. So, the valgrind output does not seem to help at all.
This should be reproducible! Please make sure you can reproduce it before providing an answer.
EDIT2 (revised code and valgrind output, removed pointer method)
Here I have changed the list from a list of pointers to a list of instances. The memory increase is confirmed.
#include <sys/resource.h> // memory management.
#include <stdio.h>
#include <iostream>
#include <iomanip>
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/core/core.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/video/background_segm.hpp"
using namespace std;
using namespace cv;
// Load frame from disk.
void readFrame(int frameNum, Mat &frame) {
// Construct filenames
Mat image;
stringstream number, filename;
number << setw(7) << setfill('0') << frameNum; // expecting over 1e10 images over the installation period.
filename << "../images/store-" << number.str() << ".jpg"; // assumes jpegs!//
cout << "Loading filename: " << filename.str() << endl;
image = imread( filename.str() );
if (image.empty() or !image.data) {
cout << "Input image empty:\n";
}
frame = image.clone();
}
// Class to hold the perceptual chunks.
class percepUnit {
public:
cv::Mat image; // percept itself
cv::Mat mask; // alpha channel
// constructor method
percepUnit(cv::Mat &ROI, cv::Mat &alpha, int ix, int iy, int iw, int ih, int area) {
image = ROI.clone();
mask = alpha.clone();
}
};
// Segment foreground from background
void segmentForeground(list<percepUnit> &percepUnitsForeground, Mat &foreground, Mat &frame) {
Mat contourImage = Mat(foreground.rows, foreground.cols, CV_8UC1, Scalar::all(0));
vector<vector<Point>> contours;
int area;
// The following causes strange spikes in memory usage:
// find contours
findContours(foreground, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
for (int idx = 0; idx < contours.size(); idx++) {
area = contourArea(contours[idx]);
if (area > 100) {
percepUnit thisUnit = percepUnit(frame, contourImage, 0, 0, 100,100, area);
percepUnitsForeground.push_back(thisUnit); // Append to percepUnits
}
}
/* The following does not:
findContours(foreground, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
for (int idx = 0; idx < contours.size(); idx++) {
area = contourArea(contours[idx]);
}*/
/* Neither does this:
for (int idx = 0; idx < 10; idx++) {
percepUnit thisUnit = percepUnit(frame, contourImage, 0, 0, 100,100, area);
percepUnitsForeground.push_back(thisUnit); // Append to percepUnits
}*/
}
int main(int argc, const char** argv)
{
int frameCount = 78298;
Mat frame, foreground;
BackgroundSubtractorMOG2 MOG2model;
list<percepUnit> scratchPercepUnitsForeground;
// add rusage stuff
struct rusage usage; // memory usage.
for(int i=0; i<= 75; i++)
{
// run full segmenter here. (background disabled)
readFrame(frameCount, frame); // was frame = readFrame();
// Only process if this frame actually loaded (non empty)
if ( not frame.empty() ) {
MOG2model(frame,foreground); // Update MOG2 model, downscale?
// before we segment again clear scratch
scratchPercepUnitsForeground.clear();
// Segment the foreground regions and generate boolImage to extract from background.
segmentForeground(scratchPercepUnitsForeground, foreground, frame);
}
frameCount++;
getrusage(RUSAGE_SELF, &usage);
cout << "DEBUG leakTest_bug_report " << i << " " << usage.ru_maxrss/1024.0 << endl;
}
return 0;
}
Here is the corresponding valgrind output:
==3562== Memcheck, a memory error detector
==3562== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3562== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3562== Command: ./leakTest
==3562==
==3562==
==3562== HEAP SUMMARY:
==3562== in use at exit: 15,024 bytes in 7 blocks
==3562== total heap usage: 795,556 allocs, 795,549 frees, 29,269,731,785 bytes allocated
==3562==
==3562== 568 bytes in 1 blocks are still reachable in loss record 1 of 7
==3562== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3562== by 0x63A720A: __fopen_internal (iofopen.c:76)
==3562== by 0xA8BC050: libjpeg_general_init (in /usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2)
==3562== by 0x400F305: call_init.part.0 (dl-init.c:85)
==3562== by 0x400F3DE: _dl_init (dl-init.c:52)
==3562== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==3562==
==3562== 2,072 bytes in 1 blocks are still reachable in loss record 2 of 7
==3562== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3562== by 0x1495F675: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x1495E4AE: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x14950888: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x149253B8: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x400F305: call_init.part.0 (dl-init.c:85)
==3562== by 0x400F3DE: _dl_init (dl-init.c:52)
==3562== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==3562==
==3562== 2,072 bytes in 1 blocks are still reachable in loss record 3 of 7
==3562== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3562== by 0x1495F675: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x1495E0EF: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x14950890: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x149253B8: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x400F305: call_init.part.0 (dl-init.c:85)
==3562== by 0x400F3DE: _dl_init (dl-init.c:52)
==3562== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==3562==
==3562== 2,072 bytes in 1 blocks are still reachable in loss record 4 of 7
==3562== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3562== by 0x1495F675: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x14971A6F: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x14950898: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x149253B8: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x400F305: call_init.part.0 (dl-init.c:85)
==3562== by 0x400F3DE: _dl_init (dl-init.c:52)
==3562== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==3562==
==3562== 2,072 bytes in 1 blocks are still reachable in loss record 5 of 7
==3562== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3562== by 0x1495F675: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x1499024F: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x149508A0: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x149253B8: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x400F305: call_init.part.0 (dl-init.c:85)
==3562== by 0x400F3DE: _dl_init (dl-init.c:52)
==3562== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==3562==
==3562== 2,072 bytes in 1 blocks are still reachable in loss record 6 of 7
==3562== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3562== by 0x1495F675: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x149610EF: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x149253B8: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.24.4)
==3562== by 0x400F305: call_init.part.0 (dl-init.c:85)
==3562== by 0x400F3DE: _dl_init (dl-init.c:52)
==3562== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==3562==
==3562== 4,096 bytes in 1 blocks are still reachable in loss record 7 of 7
==3562== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3562== by 0xA8BC067: libjpeg_general_init (in /usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2)
==3562== by 0x400F305: call_init.part.0 (dl-init.c:85)
==3562== by 0x400F3DE: _dl_init (dl-init.c:52)
==3562== by 0x40016E9: ??? (in /lib/x86_64-linux-gnu/ld-2.15.so)
==3562==
==3562== LEAK SUMMARY:
==3562== definitely lost: 0 bytes in 0 blocks
==3562== indirectly lost: 0 bytes in 0 blocks
==3562== possibly lost: 0 bytes in 0 blocks
==3562== still reachable: 15,024 bytes in 7 blocks
==3562== suppressed: 0 bytes in 0 blocks
==3562==
==3562== For counts of detected and suppressed errors, rerun with: -v
==3562== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
And yet, memory still increases unexplainably: (the overall increase from the previous plot is due to running this test in valgrind.)
(source: ekran.org)
For the same code here is the massif output: http://www.ekran.org/tmp/massif.print.leak
The massif output for the non-leak case where findContours() is not called, only the percepUnit constructor: http://www.ekran.org/tmp/massif.print.noLeak
EDIT3
It was suggested in the cross-thread (http://answers.opencv.org/question/19172/bug-increasing-memory-usage-per-iteration-when/) that I read proc rather than using the rusage method, and look at this, memory does not steadily increase: (!)
(source: ekran.org)
This does appear similar to the massif output!! So I guess I need to redo all my unit tests. Anyone have a reason I should not give up here and consider the issue rusage?
Your for-loop for your ptr-list cleanup is skipping items due to having both an increment-clause (percepIter++) and an iterator reassignment in the loop body itself (the return value from the erase() call).
In other words, you're double-incrementing your iterator, skipping every even-slotted items.
I've marked it below:
for (list<percepUnit*>::iterator percepIter = scratchPercepUnitsForeground.begin();
percepIter != scratchPercepUnitsForeground.end();
percepIter++) { // ADVANCES ITERATOR
delete *percepIter; // delete what we point to.
percepIter = scratchPercepUnitsForeground.erase(percepIter); // ADVANCES ITERATOR
}
You can address this a number of ways. For example, by removing the increment expression from your for-loop:
for (list<percepUnit*>::iterator percepIter = scratchPercepUnitsForeground.begin();
percepIter != scratchPercepUnitsForeground.end();) {
delete *percepIter; // delete what we point to.
percepIter = scratchPercepUnitsForeground.erase(percepIter);
}
Likewise, you can simply enumerate the list, then invoke the list clear() method once you've freed all the pointed-to objects.
for (list<percepUnit*>::iterator percepIter = scratchPercepUnitsForeground.begin();
percepIter != scratchPercepUnitsForeground.end();
++percepIter)
{
delete *percepIter;
}
scratchPercepUnitsForeground.clear();
Personally, I prefer the latter if I had to choose one or the other. It is, among other things, faster, and imho more readable.
But were I coding this I'd use smart pointers, which would make this completely irrelevant, as you could simply fire scratchPercepUnitsForeground.clear(); and be done with it. When the list content is cleared all the smart-pointer destructors would fire, and they in-turn would delete their objects for you. The concept is called Resource Acquisition Is Initialization, or RAII for short, and it simply means all things, including dynamic allocations, have scope-based lifetimes with automatic reclaiming of resources when scope exits. You can read more about it here.
Anyway, thats where there is definitely a leak, and a big one judging by the looks of things.
To me, the fact that your graph profiling with rusage increases monotonically is suspect.
The documentation of rusage states:
The maximum resident set size used, in kilobytes. That is, the maximum number of kilobytes of physical memory that processes used simultaneously.
Since you are likely gathering statistics for the same process, only across multiple images, the conclusion that rusage reports the maximum memory usage across all images makes sense.
Indeed, using a different profiling tool (Instruments on OS X) generates this graph showing current memory usage:
Which is quite similar to your results using proc. I would conclude that rusage is not the profiling tool you want.
I found this question after googling what seems to be the same problem (valgrind reporting "still reachable" memory with a backtrace including _dl_init and libpixman-1). I'm using Fedora 18 64-bit. I managed to reproduce the problem with an absolutely minimal C executable and .so library... the C program has just a main function that returns 0, and the .so has just one function (never actually called), which also exits immediately with 0. Yet valgrind still reports 5 unfreed blocks totalling 10,360 bytes. I decided to post this in case some other poor soul has spent a month debugging the same problem! Whatever the bug is, it seems unlikely to be caused by an executable and library, both of which do nothing.
On further investigation, the memory leak disappeared when I recompiled the executable and the.so file on Ubuntu 13.04 (GCC 4.7.3, compared to Fedora 18's GCC 4.7.2). I made no other changes so the problem definitely wasn't with my code. The fixed .so continued to be valgrind-perfect even after i transfered it back to the Fedora machine. I would suggest recompiling any/all shared objects you are using, with a newer version of gcc. As for the question of memory usage increasing over time, I don't know because my project is very small, allocates little memory and exits quickly.
Thats the Store Credit problem on Google code jam.
https://code.google.com/codejam/contest/351101/dashboard#s=p0
My code gives out a SIGSEGV after running the large test. But the answer is correct!
#include <cstdio>
#include <algorithm>
#include <vector>
using namespace std;
int ps[1000]={0};
vector<int> indice[1000];
int main() {
int cases; scanf("%d", &cases);
for(int j=1;j<=cases;j++) {
printf("Case #%d: ", j);
int c, is; scanf("%d%d", &c, &is);
for(int i=0;i<=c;i++) ps[i]=0;
for(int i=0;i<=c;i++) indice[i].clear();
for (int i = 0; i < is; i++) {
int it; scanf("%d", &it);
indice[it].push_back(i+1);
ps[it]=1;
if (c-it>0&&ps[c-it]) {
int a, b;
a = indice[it][0];
b = indice[c-it][0];
if(c==2*it&&indice[it].size()>1) {
b=indice[it][1];
}
if (a!=b) {
printf("%d %d\n", min(a,b),max(a,b));
}
}
}
}
return 0;
}
So I use valgrind to find out what's going on .. but it seems that it's not my problem.
==17599== Invalid free() / delete / delete[] / realloc()
==17599== at 0x4C2A4BC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==17599== by 0x401669: __gnu_cxx::new_allocator<int>::deallocate(int*, unsigned long) (new_allocator.h:98)
==17599== by 0x4013CD: std::_Vector_base<int, std::allocator<int> >::_M_deallocate(int*, unsigned long) (stl_vector.h:156)
==17599== by 0x400F60: std::_Vector_base<int, std::allocator<int> >::~_Vector_base() (stl_vector.h:142)
==17599== by 0x400D8D: std::vector<int, std::allocator<int> >::~vector() (stl_vector.h:351)
==17599== by 0x400C48: __tcf_0 (a.cpp:6)
==17599== by 0x5383900: __run_exit_handlers (exit.c:78)
==17599== by 0x5383984: exit (exit.c:100)
==17599== by 0x5369773: (below main) (libc-start.c:258)
==17599== Address 0x1 is not stack'd, malloc'd or (recently) free'd
==17599==
==17599==
==17599== HEAP SUMMARY:
==17599== in use at exit: 128 bytes in 1 blocks
==17599== total heap usage: 4,527 allocs, 4,527 frees, 113,664 bytes allocated
==17599==
==17599== LEAK SUMMARY:
==17599== definitely lost: 0 bytes in 0 blocks
==17599== indirectly lost: 0 bytes in 0 blocks
==17599== possibly lost: 0 bytes in 0 blocks
==17599== still reachable: 128 bytes in 1 blocks
==17599== suppressed: 0 bytes in 0 blocks
==17599== Rerun with --leak-check=full to see details of leaked memory
==17599==
==17599== For counts of detected and suppressed errors, rerun with: -v
==17599== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
I'm so confused... Could anybody tell me what's going on? I'm a newbie of C++ ..
Thank you very much.
As I understand valgrind, it can't detect that you are writing outside of the bounds of arrays that are staticly allocated. So let's allocate them on the heap.
vector<int> *indice = new vector<int>[1000];
int *ps = new int[1000];
Then, you will see errors come out of valgrind. Including:
==7168== Invalid read of size 8
==7168== at 0x4008D6: main (stl_vector.h:735)
==7168== Address 0x4c39e10 is 8 bytes after a block of size 24,008 alloc'd
==7168== at 0x4A07152: operator new[](unsigned long) (vg_replace_malloc.c:363)
==7168== by 0x400791: global constructors keyed to indice (foo.cc:6)
==7168== by 0x400C35: ??? (in /tmp/foo)
==7168== by 0x4005F2: ??? (in /tmp/foo)
==7168==
==7168== Invalid read of size 8
==7168== at 0x4008DA: main (stl_vector.h:735)
==7168== Address 0x4c39e18 is 16 bytes after a block of size 24,008 alloc'd
==7168== at 0x4A07152: operator new[](unsigned long) (vg_replace_malloc.c:363)
==7168== by 0x400791: global constructors keyed to indice (foo.cc:6)
==7168== by 0x400C35: ??? (in /tmp/foo)
==7168== by 0x4005F2: ??? (in /tmp/foo)
==7168==
And using gdb, I can see that the SIGSEGV occurs when you access indice[1433], which is outside of the bounds of indice.
I also imagine that your actual issue is that for the large dataset, the variables bounds are listed as:
N = 50
3 ≤ I ≤ 2000
Are you sure you shouldn't be allocating 2001 elements, instead of 1000?