I have Python extensions implemented on C++ classes. I don't have a C++ target to run valgrind with. I want to use valgrind for memory check.
Can I use valgrind with Python?
Yes, you can use valgrind with Python. You just need to use the valgrind suppression file provided by the Python developers, so you don't get a bunch of false positives due to Python's custom memory allocation/reallocation functions.
The valgrind suppression file can be found here: http://svn.python.org/projects/python/trunk/Misc/valgrind-python.supp
IMPORTANT: You need to uncomment the lines for PyObject_Free and PyObject_Realloc in the suppression file*.
The recommended usage syntax is:
$ valgrind --tool=memcheck --suppressions=valgrind-python.supp \
python -E -tt ./my_python_script.py
See also this README file from the Python SVN repo which describes the different ways of using Python with valgrind:
http://svn.python.org/projects/python/trunk/Misc/README.valgrind
* - Alternatively, you can recompile Python with PyMalloc disabled, which allows you to catch more memory leaks that won't show up if you just suppress PyMalloc.
In Python 2.7 and 3.2 there is now a --with-valgrind compile-time flag that allows the Python interpreter to detect when it runs under valgrind and disables PyMalloc. This should allow you to more accurately monitor your memory allocations than otherwise, as PyMalloc just allocates memory in big chunks.
Yes you can: you do have a target to run valgrind with -- it's the python interpreter itself:
valgrind python foo.py
However, the results of above may not be very satisfactory -- Python is built in opt mode and with a special malloc, which may drown you in false positives.
You'll likely get better results by first building a debug version of Python. Start here.
Related
I have an OCaml program that worked fine on Ubuntu 16 but when recompiled and run on Ubuntu 20 I get the following error:-
$ ocamldebug ./linearizer
OCaml Debugger version 4.08.1
(ocd) r
Loading program... done.
Time: 89534
Program end.
Uncaught exception: Sys_error "Illegal seek"
(ocd) b
Time: 89533 - pc: 624888 - module Netaccel_link
No source file for Netaccel_link.
I thought this was due to missing dev libraries but:-
$ sudo apt install libocamlnet-ocaml-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libocamlnet-ocaml-dev is already the newest version (4.1.6-1build6).
0 upgraded, 0 newly installed, 0 to remove and 20 not upgraded.
What setup step am I missing on Ubuntu 20?
This looks like a regression bug in libocamlnet and you should report an issue there or, I am a bit pessimistic that you will get any response, you can try to debug the issue yourself.
The problem that you are facing has nothing to do with missing libraries (they will be reported during installation or, if the package is broken, end up in linker errors). It may result, however, from some misconfiguration of the system. If that is true, then you're lucky as you can fix it yourself.
I will give you some advice that might help you in debugging this issue. For more, please try using discuss.ocaml.org as a more suitable media (SO doesn't favor this kind of a discussion and we might get deleted by admins).
The illegal seek exception is thrown when the seek operation is applied on a non-regular file, aka ESPIPE Unix error. So check your inputs. It could be that what was previously regarded as a file in Ubuntu is now a pipe or a socket.
Try to use ltrace or strace to pinpoint the culprit e.g.,
ltrace ./linearizer
or, if it overwhelms you, try strace
strace ./linearizer
Instead of using ocamldebug you can use plain gdb. You can use gdb's interfaces to provide the path to the source code (though most likely it won't work since ocamlnet is not compiled with debug information). I believe that it will give you a more meaningful backtrace.
Instead of using the system installation try using opam. Install your dependencies with opam and try older versions as well as newer versions of the OCaml compiler. Also, try different versions of ocamlnet. Ideally, try to reproduce the environment that used to work for you.
When nothing else works, you can use objdump -d and look at the disassembly of your binary. OCaml is using a pretty readable and intuitive name mangling scheme (<module_name>__<function_name>_<uid>), so you can easily find the source code (search for <module_name>.ml file and look for the <function_name> there)
Finally, just use docker or any other container to run your application. Consider switching from ocamlnet to something more modern and supported.
I am writing an R package that includes C++ code, written by an external developer who is unable to help out.
The C++ code currently appears to have a memory leak: R's memory usage keeps increasing when the C++ code is run, and is not released until R is quit. It is my task to neutralize this leak.
Because I am using Windows, and calling the C++ code through R, it is not clear how best to track down this leak. My cunning plan was to use valgrind in a Linux environment on Travis CI, but this finds no problems.
What is the best way to track down memory leaks?
I have had partial success by adding a separate call to R with valgrind to my .travis.yml.
addons:
apt:
packages: valgrind
after_success:
- R -d "valgrind --leak-check=full --track-origins=yes" --vanilla < tests/testthat/valgrind.R
Ideally I'd've run tests/testthat.R, but because R -d runs interactively, I've had to create a separate file tests/testthat/valgrind.R for tests:
library(testthat)
# Load the package
library(pkgload)
load_all()
# This may run during build in root working directory,
# then again with R -d from tests wd.
if (dir.exists('testthat')) setwd('testthat/tests')
testFiles <- list.files(pattern = 'test\\-.*\\.R', full.names= TRUE)
# Test files using expect_doppleganger will fail in interactive mode.
# Remove them.
lapply(testFiles[-c(3, 6)], source)
This doesn't feel like an optimal solution... but it is sufficient for my immediate needs.
I need to set LD_LIBRARY_PATH, LD_PRELOAD and some environment variables for a process while running and detect memory leaks with Valgrind.
Can anyone suggest a way to set or pass these variable for a process in valgrind?.
I've run into a similar issue, trying to run valgrind on programs that need libraries incompatible with the ones valgrind uses, and have been using:
valgrind --trace-children=yes env LD_LIBRARY_PATH=your_library_path OTHER_VAR=foo your_program arg1 arg2...
env sets up the environment and then execs your_program. We need to pass the --trace-children=yes argument to valgrind in order for it to continue to trace through the exec syscall. Without --trace-children=yes set, valgrind will stop tracing at the exec and you won't get any useful output from valgrind on your_program.
One potential downside to this approach is that valgrind might report any memory issues in env. I haven't seen any false positives from this source (env is not a very complicated program), but it could happen.
I haven't tried this with LD_PRELOAD though (it hasn't come up for my use-case yet). Valgrind does set LD_PRELOAD, so you might have to do something like:
valgrind --trace-children=yes env LD_PRELOAD=$LD_PRELOAD:your_preload your_program
What is wrong with the standard mechanisms? These include:
LD_LIBRARY_PATH=$new_libpath LD_PRELOAD=$new_preload OTHERVAR=otherval valgrind your.program arg1 …
Or:
env LD_LIBRARY_PATH=$new_libpath \
LD_PRELOAD=$new_preload \
OTHERVAR=otherval \
valgrind ./your.program arg1 …
Or:
export LD_LIBRARY_PATH=$new_libpath
export LD_PRELOAD=$new_preload
export OTHERVAR=otherval
valgrind ./your.program arg1 …
The advantage of the first two mechanisms is that it doesn't affect the working environment of your shell. The advantage of the last mechanism is that it does affect the working environment of your shell (which makes it easier to run valgrind the next time — you don't have to remember to find the command with the environment in your history).
in C/C++, I can include valgrind headers to know at runtime whether or not my software is running on valgrind :
#include <valgrind/valgrind.h>
bool RunningOnValgrind()
{
return RUNNING_ON_VALGRIND ? true : false;
}
This is documented in the valgrind manual.
I would like to be able to know if the valgrind I am being run on supports AVX instructions. How do I write a function that returns this information ?
From valgrind release notes, I known that these are supported from version 3.8 onwards. Hence one solution would be to spawn a process to execute valgrind --version and then parse the output but there must be a better way.
If you look in valgrind/valgrind.h, you see you can check valgrind version number this way after including valgrind.h:
#if defined(__VALGRIND_MAJOR__) && defined(__VALGRIND_MINOR__) \
&& (__VALGRIND_MAJOR__ > 3 \
|| (__VALGRIND_MAJOR__ == 3 && __VALGRIND_MINOR__ >= 8))
/* code to say avx is supported */
#endif
This has many limitations, however: it assumes you are using a globally-installed version, and that your path isn't pointing to some personal, user-built valgrind that isn't in the default location (which I have done) and it also assumes that you are building on the machine where you will run valgrind, and not shipping around a pre-built executable (which in my experience is something that happens often) so you can't rely on that.
With that in mind, spawning a sub-process with valgrind --version and checking the output may truly be the best alternative.
You can use VALGRIND_MAJOR to detect the version in runtime, or use --version flag to get the exact version.
I'm trying to profile a C++ application, that I did not write, to get a sense for where the major computation points are. I'm not a C++ expert and even less so C++ debugging/profiling expert. I believe I am running into a (common?) problem with dynamic libraries.
I compile link to Google CPU Profiler using (OS X, G++):
env LIBS=-lprofiler ./configure
make
make install
I then run profile the installed application (jags) with:
env CPUPROFILE=./jags.prof /usr/local/bin/jags regression.cmd
pprof /usr/local/bin/jags jags.prof
Unfortunately, I get the error:
pprof /usr/local/bin/jags jags.prof Can't exec "objdump":
No such file or directory at /usr/local/bin/pprof line 2833.
objdump /System/Library/Frameworks/Accelerate.framework/Versions/A/
Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib: No such file or directory
The program dynamically links to libLAPACK.dylib. So prof does not seem to understand it (?). I thought about trying to statically link, but the documents associated with the program say that it is impossible to statically link in LAPACK or BLAS (two required libraries).
Is there a way to have the profiler ignore libLAPACK? I'm okay if it doesn't sample within libLAPACK. Or how might I get profiling to work?
This error was caused by jags being a shell script, that subsequently called profilable code.
pprof /usr/local/bin/REAL_EXEC jags.prof
fixes the problem.
I don't see a clean way to do it, but maybe there's a hacky workaround -- what happens if you hack the pprof perl script (or better a copy thereof;-), line 2834, so that instead of calling error it emits the message and then does return undef;?
If you're profiling on OSX, the Shark tool is really great as well. It's very simple to use, and has worked out of the box for me when I've tried it.