Closed. This question needs to be more focused. It is not currently accepting answers.
Closed 8 years ago.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
What are your best tips for debugging Python?
Please don't just list a particular debugger without saying what it can actually do.
Related
What are good ways to make my Python code run first time? - This discusses minimizing errors
PDB
You can use the pdb module, insert pdb.set_trace() anywhere and it will function as a breakpoint.
>>> import pdb
>>> a="a string"
>>> pdb.set_trace()
--Return--
> <stdin>(1)<module>()->None
(Pdb) p a
'a string'
(Pdb)
To continue execution use c (or cont or continue).
It is possible to execute arbitrary Python expressions using pdb. For example, if you find a mistake, you can correct the code, then type a type expression to have the same effect in the running code
ipdb is a version of pdb for IPython. It allows the use of pdb with all the IPython features including tab completion.
It is also possible to set pdb to automatically run on an uncaught exception.
Pydb was written to be an enhanced version of Pdb. Benefits?
http://pypi.python.org/pypi/pudb, a full-screen, console-based Python debugger.
Its goal is to provide all the niceties of modern GUI-based debuggers in a more lightweight and keyboard-friendly package. PuDB allows you to debug code right where you write and test it – in a terminal. If you've worked with the excellent (but nowadays ancient) DOS-based Turbo Pascal or C tools, PuDB's UI might look familiar.
Nice for debugging standalone scripts, just run
python -m pudb.run my-script.py
If you are using pdb, you can define aliases for shortcuts. I use these:
# Ned's .pdbrc
# Print a dictionary, sorted. %1 is the dict, %2 is the prefix for the names.
alias p_ for k in sorted(%1.keys()): print "%s%-15s= %-80.80s" % ("%2",k,repr(%1[k]))
# Print the instance variables of a thing.
alias pi p_ %1.__dict__ %1.
# Print the instance variables of self.
alias ps pi self
# Print the locals.
alias pl p_ locals() local:
# Next and list, and step and list.
alias nl n;;l
alias sl s;;l
# Short cuts for walking up and down the stack
alias uu u;;u
alias uuu u;;u;;u
alias uuuu u;;u;;u;;u
alias uuuuu u;;u;;u;;u;;u
alias dd d;;d
alias ddd d;;d;;d
alias dddd d;;d;;d;;d
alias ddddd d;;d;;d;;d;;d
Logging
Python already has an excellent built-in logging module. You may want to use the logging template here.
The logging module lets you specify a level of importance; during debugging you can log everything, while during normal operation you might only log critical things. You can switch things off and on.
Most people just use basic print statements to debug, and then remove the print statements. It's better to leave them in, but disable them; then, when you have another bug, you can just re-enable everything and look your logs over.
This can be the best possible way to debug programs that need to do things quickly, such as networking programs that need to respond before the other end of the network connection times out and goes away. You might not have much time to single-step a debugger; but you can just let your code run, and log everything, then pore over the logs and figure out what's really happening.
EDIT: The original URL for the templates was: http://aymanh.com/python-debugging-techniques
This page is missing so I replaced it with a reference to the snapshot saved at archive.org: http://web.archive.org/web/20120819135307/http://aymanh.com/python-debugging-techniques
In case it disappears again, here are the templates I mentioned. This is code taken from the blog; I didn't write it.
import logging
import optparse
LOGGING_LEVELS = {'critical': logging.CRITICAL,
'error': logging.ERROR,
'warning': logging.WARNING,
'info': logging.INFO,
'debug': logging.DEBUG}
def main():
parser = optparse.OptionParser()
parser.add_option('-l', '--logging-level', help='Logging level')
parser.add_option('-f', '--logging-file', help='Logging file name')
(options, args) = parser.parse_args()
logging_level = LOGGING_LEVELS.get(options.logging_level, logging.NOTSET)
logging.basicConfig(level=logging_level, filename=options.logging_file,
format='%(asctime)s %(levelname)s: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
# Your program goes here.
# You can access command-line arguments using the args variable.
if __name__ == '__main__':
main()
And here is his explanation of how to use the above. Again, I don't get the credit for this:
By default, the logging module prints critical, error and warning messages. To change this so that all levels are printed, use:
$ ./your-program.py --logging=debug
To send log messages to a file called debug.log, use:
$ ./your-program.py --logging-level=debug --logging-file=debug.log
It is possible to print what Python lines are executed (thanks Geo!). This has any number of applications, for example, you could modify it to check when particular functions are called or add something like ## make it only track particular lines.
code.interact takes you into a interactive console
import code; code.interact(local=locals())
If you want to be able to easily access your console history look at: "Can I have a history mechanism like in the shell?" (will have to look down for it).
Auto-complete can be enabled for the interpreter.
ipdb is like pdb, with the awesomeness of ipython.
print statements
Some people recommend a debug_print function instead of print for easy disabling
The pprint module is invaluable for complex structures
the obvious way to debug a script
python -m pdb script.py
useful when that script raises an exception
useful when using virtualenv and pdb command is not running with the venvs python version.
if you don't know exactly where that script is
python -m pdb ``which <python-script-name>``
PyDev
PyDev has a pretty good interactive debugger. It has watch expressions, hover-to-evaluate, thread and stack listings and (almost) all the usual amenities you expect from a modern visual debugger. You can even attach to a running process and do remote debugging.
Like other visual debuggers, though, I find it useful mostly for simple problems, or for very complicated problems after I've tried everything else. I still do most of the heavy lifting with logging.
If you are familiar with Visual Studio, Python Tools for Visual Studio is what you look for.
Winpdb is very nice, and contrary to its name it's completely cross-platform.
It's got a very nice prompt-based and GUI debugger, and supports remote debugging.
In Vim, I have these three bindings:
map <F9> Oimport rpdb2; rpdb2.start_embedded_debugger("asdf") #BREAK<esc>
map <F8> Ofrom nose.tools import set_trace; set_trace() #BREAK<esc>
map <F7> Oimport traceback, sys; traceback.print_exception(*sys.exc_info()) #TRACEBACK<esc>
rpdb2 is a Remote Python Debugger, which can be used with WinPDB, a solid graphical debugger. Because I know you'll ask, it can do everything I expect a graphical debugger to do :)
I use pdb from nose.tools so that I can debug unit tests as well as normal code.
Finally, the F7 mapping will print a traceback (similar to the kind you get when an exception bubbles to the top of the stack). I've found it really useful more than a few times.
Defining useful repr() methods for your classes (so you can see what an object is) and using repr() or "%r" % (...) or "...{0!r}..".format(...) in your debug messages/logs is IMHO a key to efficient debugging.
Also, the debuggers mentioned in other answers will make use of the repr() methods.
Getting a stack trace from a running Python application
There are several tricks here. These include
Breaking into an interpreter/printing a stack trace by sending a signal
Getting a stack trace out of an unprepared Python process
Running the interpreter with flags to make it useful for debugging
If you don't like spending time in debuggers (and don't appreciate poor usability of pdb command line interface), you can dump execution trace and analyze it later. For example:
python -m trace -t setup.py install > execution.log
This will dump all source line of setup.py install execution to execution.log.
To make it easier to customize trace output and write your own tracers, I put together some pieces of code into xtrace module (public domain).
When possible, I debug using M-x pdb in emacs for source level debugging.
There is a full online course called "Software Debugging" by Andreas Zeller on Udacity, packed with tips about debugging:
Course Summary
In this class you will learn how to debug programs systematically, how
to automate the debugging process and build several automated
debugging tools in Python.
Why Take This Course?
At the end of this course you will have a solid understanding about
systematic debugging, will know how to automate debugging and will
have built several functional debugging tools in Python.
Prerequisites and Requirements
Basic knowledge of programming and Python at the level of Udacity
CS101 or better is required. Basic understanding of Object-oriented
programming is helpful.
Highly recommended.
if you want a nice graphical way to print your call stack in a readable fashion, check out this utility: https://github.com/joerick/pyinstrument
Run from command line:
python -m pyinstrument myscript.py [args...]
Run as a module:
from pyinstrument import Profiler
profiler = Profiler()
profiler.start()
# code you want to profile
profiler.stop()
print(profiler.output_text(unicode=True, color=True))
Run with django:
Just add pyinstrument.middleware.ProfilerMiddleware to MIDDLEWARE_CLASSES, then add ?profile to the end of the request URL to activate the profiler.
Related
I am trying to do a stackoverflow for a course at university. The binary I am to exploit has a canary, however, there is a way to leak that canary to stdout. The canary of course consists of some random bytes so I can't just read them from the string that the program outputs to stdout.
For this reason I am using the python and pwntools like p.recv(timeout = 0.01).encode("hex").
(I'm using pwntools only because I don't know another way to read the output in hex format, if there is an easier way I can of course use something else)
This works more or less works as expected, I manage to write the memory area that is past the canary. However, I get a segfault, so I obviously have some problem with the stackoverflow I am causing. I need a way of debugging this, like seeing the stack after I provide the input that causes the stackoverflow.
And now without any further ado the actual question: Can I debug a process that I started with pwntools (like process("./myprog")) in GDB or some other program that can show me the content of the stack?
I already tried getting the pid in python and using gdb attach to attach to that pid, but that didn't work.
Note: The binary I am trying to exploit has the guid set. Don't know if that matters tho.
You can use the pwnlib.gdb to interface with gdb.
You can use the gdb.attach() function:
From the docs:
bash = process('bash')
# Attach the debugger
gdb.attach(bash, '''
set follow-fork-mode child
break execve
continue
''')
# Interact with the process
bash.sendline('whoami')
or you can use gdb.debug():
# Create a new process, and stop it at 'main'
io = gdb.debug('bash', '''
# Wait until we hit the main executable's entry point
break _start
continue
# Now set breakpoint on shared library routines
break malloc
break free
continue
''')
# Send a command to Bash
io.sendline("echo hello")
# Interact with the process
io.interactive()
The pwntools template contains code to get you started with debugging with gdb. You can create the pwntools template by running pwn template ./binary_name > template.py. Then you have to add the GDB arg when you run template.py to debug: ./template.py GDB.
If you get [ERROR] Could not find a terminal binary to use., you might need to set context.terminal before you use gdb.
If you're using tmux, the following will automatically open up a gdb debugging session in a new horizontally split window:
context.terminal = ["tmux", "splitw", "-h"]
And to split the screen with the new gdb session window vertically:
context.terminal = ["tmux", "splitw", "-v"]
(Note: I never got this part working, so idk if it'll work. Tell me if you get the gdb thing working).
(To use tmux, install tmux on your machine, and then just type tmux to start it. Then type python template.py GDB.
If none of the above works, then you can always just start your script, use ps aux, find the PID, and then use gdb -p PID to attach to the running process.
When debuging a simple program in gdb, I want to continue the execution automatically after hitting breakpoints. As far as I know, there are two methods to accomplish it:
1) use hook-stop.
define hook-stop
continue
end
But it seems the hook-stop is trigged only once. When another breakpoint is hit next time, the execution still stops.
2) use gdb.events.stop.connect().
def handle_stop_event(event):
if isinstance(event, gdb.BreakpointEvent):
gdb.execute('continue')
gdb.events.stop.connect(handle_stop_event)
This method works well. But if there are too many breakpoints been hit, an error "Fatal Python error: Cannot recover from stack overflow." occurs.
It seems because of recursive call. I'm wondering why the gdb.execute('continue') would cause this issue.
I searched online and still didn't find a solution.
PS: gdb version 7.11.1 on Ubuntu 16.04
Any advice would be appreciate! Thanks in advance.
It seems, continue inside hook-stop doesn't work properly. Have you seen this question I posted yesterday?
I think, the best approach here is writing a convenience function in python and setting a conditional breakpoint. Or using commands — see the "Breakpoint Command Lists" section of the GDB user manual.
Here's how do to it (also described in the manual).
The python module:
import gdb
class should_skip_f(gdb.Function):
def __init__ (self):
super (should_skip_f, self).__init__("should_skip")
def invoke(self):
return True # Your condition here
should_skip_f()
(gdb) b <your target> if !$should_skip()
Or add the condition to existing breakpoints with
(gdb) condition <BNUM> !$should_skip()
The only downside is that you have to set the condition for each breakpoint individually, but that's scriptable. Also, I think, the commands syntax allows you to add commands to a list of breakpoints at once.
'commands [LIST...]'
'... COMMAND-LIST ...'
'end'
Specify a list of commands for the given breakpoints. The commands
themselves appear on the following lines. Type a line containing
just 'end' to terminate the commands.
As for the recursion — yeah, that's a bad "design" of a debugger script (if one should talk about design of one-off throwaway things). You can examine what happens there if you extend your python script like so
import inspect
...
def handle_stop_event(event):
...
print(len(inspect.stack())) # Or you can print the frames themselves...
The Python interpreter doesn't know that execution does not return from gdb.execute("continue"), so the Python stack frames for invocations of this function are never destroyed.
You can increase max stack size for the interpreter, but like I said, this script does not seem like the best solution to me.
First of all, I'm quite new at Python, Stackoverflow and programming in general so please forgive any formal errors I might have made as I'm still trying to grasp many of the required conceptual programming protocols.
Here's the issue:
I'm trying to work around a specific, seemingly simple problem I've been having when using Tkinter: Whenever I'm fiddling with some code that confuses me, it generally takes many attempts until I finally find a working solution. So I write some code, run it, get an error, make some changes, run it again, get another error, change it again... and so on until a working result is achieved.
When the code finally works, I unfortunately also get additional Tkinter main windows popping up for every failed run I've executed. So if I've made, say, 20 changes before I eventually achieve working code, 20 additional Tkinter windows pop up. Annoying...
Now, I was thinking that maybe handling the exceptions with try/except might avoid this, but I'm unsure how to accomplish this properly.
I've been looking for a solution but can't seem to find a post that addresses this issue. I'm actually not really sure how to formulate the problem correctly... Anybody has some advice concerning this?
The following shows a simple but failed attempt of how I'm trying to circumvent this. The code works as is, but if you make a little typo in the code, run it a couple of times, then undo the typo and run the code again, you'll get multiple Tkinter windows which is what I'm trying to avoid.
Any help is, of course, appreciated...
(btw, I'm using Python 2.7.13.)
import Tkinter as tk
class App(tk.Frame):
def __init__(self, parent):
tk.Frame.__init__(self)
self.root = parent
self.canvas = tk.Canvas(self)
self.canvas.pack(expand=1,fill='both')
self.bindings()
def click(self,e):
print 'clicked'
def bindings(self):
self.root.bind('<1>',self.click)
def main():
root = tk.Tk()
app = App(root)
app.pack()
root.mainloop()
if __name__ == '__main__':
try:
main()
except:
print 'Run failed...'
Alright, great. The issue has indeed nothing to do with Tkinter or Python but with the IDE itself. Thank you Ethan for pointing that out.
PyScripter has several modes or engines. I've been running scripts with its internal engine which is faster but does not reinitialize with every run. I believe this causes the problem. The remote engine, on the other hand, does reinitialize with every run. This avoids the failed run popups.
A more in-depth explanation from the PyScripter manual below:
Python Engines:
Internal
It is faster than the other options however if there are problems with
the scripts you are running or debugging they could affect the
reliability of PyScripter and could cause crashes. Another limitation
of this engine is that it cannot run or debug GUI scripts nor it can
be reinitialized.
Remote
This the default engine of PyScripter and is the recommended engine
for most Python development tasks. It runs in a child process and
communicates with PyScripter using rpyc. It can be used to run and
debug any kind of script. However if you run or debug GUI scripts you
may have to reinitialize the engine after each run.
Remote Tk
This remote Python engine is specifically created to run and debug
Tkinter applications including pylab using the Tkagg backend. It also
supports running pylab in interactive mode. The engine activates a
Tkinter mainloop and replaces the mainloop with a dummy function so
that the Tkinter scripts you are running or debugging do not block the
engine. You may even develop and test Tkinter widgets using the
interactive console.
Remote Wx
This remote Python engine is specifically created to run and debug
wxPython applications including pylab using the WX and WXAgg backends.
It also supports running pylab in interactive mode. The engine
activates a wx MainLoop and replaces the MainLoop with a dummy
function so that the wxPython scripts you are running or debugging do
not block the engine. You may even develop and test wxPython Frames
and Apps using the interactive console. Please note that this engine
prevents the redirection of wxPython output since that would prevent
the communication with Pyscripter.
When using the Tk and Wx remote engines you can of course run or debug
any other non-GUI Python script. However bear in mind that these
engines may be slightly slower than the standard remote engine since
they also contain a GUI main loop. Also note that these two engines
override the sys.exit function with a dummy procedure.
So I am relatively new to coding so please forgive improper vocab. What I am basically trying to do is create a script for, or perhaps enter commands into, GDB so that it can run my code with the input file of a test case over and over. Basically, I am working on a project right now that makes heavy usage of semaphores and mutexes, and somewhere, every once in a blue moon, my code breaks due to race conditions. If I could have gdb run my test case continuously until my code reached a seg fault, this would be ideal.
PS- Please be specific as to what I must do, I am not great at dissecting answers that have heavy technical answers.
Thank You!
The simplest solution is expect script. Expect is a program to automate interactions with programs that expose a text terminal interface.
Examples are available at http://en.wikipedia.org/wiki/Expect
The script should be like
#!/usr/bin/expect
# start gdb
spawn gdb yourprogram
while {1} {
# wait for gdb to start, expect the (gdb) to appear
expect "(gdb)"
# send command to run your program
send "run your_args\n"
expect {
"Program exited normally." {continue} # just run again
"(Some error message)" {interact} # start to debug
}
}
You can use GDB scripts in order to automate your GDB sessions.The GDB macro coding language consists of gdb commands along with basic looping statements and conditional statements.
You can find information about it here
http://www.adacore.com/adaanswers/gems/gem-119-gdb-scripting-part-1/
What are the best ways to automate a GDB debugging session?
I'm adding some features I find useful to my GDB startup script. A few of the startup commands apply only to "live" targets, or have components that make sense only with live targets. I'd like to be able to test for the presence (or absence) of a core file, and skip or amend these commands as appropriate.
I looked around in the Python API, but couldn't find anything that tells me whether an inferior is a core file or a live program. I'm fine with a scripting solution that works in either GDB itself or in the Python GDB scripting interface.
It doesn't look like there is a way to do that.
I'd expect an attribute on gdb.Inferior, but there isn't one.
File a feature request in GDB bugzilla.
info proc status returns "unable to handle request" for core files, whereas for a live process it returns several lines, the first of which looks like: "process 1234".
You can run that command and compare its first output line against that string using the execute_output() function from here: https://github.com/crossbowerbt/GDB-Python-Utils/blob/master/gdb_utils.py