My program has 100 threads, most of which are idle and share a very well defined backtrace when they are idle. Most of the time I am only interested in the threads that are not idle and therefore do not have the "common" backtrace. I thought using a gdb script would be a good way to do this.
define backtraces
thread apply all bt
end
This script will simply print all the backtraces. Is there a way to store this output into a variable that I can then process, prune, and display only the relevant backtraces?
I naively tried:
define backtraces
set $bts = thread apply all bt
// do whatever processing here
end
But that fails with following as expected:
No symbol "thread" in current context.
Is there a better way to do this? Or good tutorials on how to power script in gdb?
Is there a better way to do this?
You would need to use Python scripting to achieve your desired result.
The filtering backtrace is probably a good start.
Using the links in Employed Russian's answer I was able to get something working. Here it is for posterity since it was completely non-obvious.
import gdb
# This loops through all the Thread objects in the process
for thread in gdb.selected_inferior().threads():
# This is equivalent to 'thread X'
thread.switch()
print "Thread %s" % thread.num
# Just execute a raw gdb command
gdb.execute('bt')
framesNames = []
f = gdb.newest_frame()
while f is not None:
framesNames.append(gdb.Frame.name(f))
f = gdb.Frame.older(f)
# do something with the name of each frame
If this were in a file named traces.py, then from gdb you can just execute the python:
source traces.py
There are other ways to invoke this python script too.
This is what I came out with based on JaredC wrote:
import gdb
class FilteredThreadBacktraceCommand(gdb.Command):
"""
Implements a command that allows printing the backtrace only for threads
that do not have given frames.
The class must be instantiated with a list of the frames to ignore. If the
stack for a thread has any of those frames then the stack for that thread
will not be printed (a visual dot will be printed to show a thread was just
ignored and provide a notion of progress).
"""
def __init__(self, ignored_frames):
super (FilteredThreadBacktraceCommand, self).__init__ ("fbt", gdb.COMMAND_STACK)
self.ignored_frames = ignored_frames
def invoke(self, arg, from_tty):
args = gdb.string_to_argv(arg)
if len(args) != 0:
gdb.write("ERROR: invalid number of arguments.\n")
return
# This loops through all the Thread objects in the process
for thread in gdb.selected_inferior().threads():
# This is equivalent to 'thread X'
thread.switch()
f = gdb.newest_frame()
frames = []
invalid_thread = False
while f is not None:
if any(ignored_frame in f.name() for ignored_frame in self.ignored_frames):
invalid_thread = True
break
frames.append(f)
f = gdb.Frame.older(f)
if invalid_thread:
# Visual effect as iterating frames might take a while
sys.stdout.write('.')
sys.stdout.flush()
continue
print "\nThread %s:" % thread.num
for i in range(len(frames)):
f = frames[i]
funcInfo = f.function()
printStr = "#{} in {}".format(i, f.name())
if f.function():
printStr += " at {}:{}".format(funcInfo.symtab.filename, funcInfo.line)
else:
printStr += " at (???)"
print(printStr)
FilteredThreadBacktraceCommand(["os::PlatformEvent::park"])
You can provide the frames that you want to ignore to the class on creation. One could also make this generic so as to provide a name to the command to register as well so that you could have different commands with different filtering.
Related
Parsing the data from Wikipedia takes an unacceptably long time. I want to do instead of one thread\process, at least 5. After googling I found that in Python 3.5 there is async for.
Below is a "very short" version of the current "synced" code to show the whole proccess (with comments to quickly understand what the code does).
def update_data(region_id=None, country__inst=None, upper_region__inst=None):
all_ids = []
# Get data about countries or regions or subregions
countries_or_regions_dict = OSM().get_countries_or_regions(region_id)
# Loop that I want to make async
for osm_id in countries_or_regions_dict:
names = countries_or_regions_dict[osm_id]['names']
if 'wiki_uri' in countries_or_regions_dict[osm_id]:
wiki_uri = countries_or_regions_dict[osm_id]['wiki_uri']
# PARSER: From Wikipedia gets translations of countries or regions or subregions
translated_names = Wiki().get_translations(wiki_uri, osm_id)
if not region_id: # Means it is country
country__inst = Countries.objects.update_or_create(osm_id=osm_id,
defaults={**countries_regions_dict[osm_id]})[0]
else: # Means it is region\subregion (in case of recursion)
upper_region__inst = Regions.objects.update_or_create(osm_id=osm_id,
country=country__inst,
region=upper_region__inst,
defaults={**countries_regions_dict[osm_id]})[0]
# Add to DB translated names from wiki
for lang_code in names:
###
# RECURSION: If country has regions or region has subregions, start recursion
if 'divisions' in countries_or_regions_dict[osm_id]:
regions_list = countries_or_regions_dict[osm_id]['divisions']
for division_id in regions_list:
all_regions_osm_ids = update_osm(region_id=division_id, country__inst=country__inst,
upper_region__inst=upper_region__inst)
all_ids += all_regions_osm_ids
return all_ids
I realized that I need to change the def update_data to async def update_data and accordingly for osm_id in countries_or_regions_dict to async for osm_id in countries_or_regions_dict,
but I could not find the information whether is it necessary to use get_event_loop() in my case and where?, and how\where to specify how many iterations of the loop can be run simultaneously? Could someone help me please to make the loop for asynchronous?
asyncio module doesn't create multiple threads/process, it run code in one thread, one process, but can handle situations, with I/O blocks (if you wrote code special way). Read, when you should use asyncio.
As soon as your code have synchronous nature, I would suggest to use threads instead of asyncio. Create ThreadPoolExecutor and use it to parse Wiki in multiple threads.
I have the following problem with this easy script:
from Tkinter import *
root = Tk()
while 1:
pass
I think, after the 2nd line everyone would expect a Tkinter window would appear. But it does not!
If I put this line into the Python console (without the endless-while-loop), it works.
[I wanted to add an image here, but as I'm new I'm to allowed to :-(]
But running the script (double-clicking on the *.py file in the Windows Explorer) results only in an empty Python console!
Background:
Actually I want to use Snack for Python. This is based on Tkinter. That means I have to create an Tk() instance first. Everything works fine in the Python console. But I want to write a bigger program with at least one Python script thus I cannot type the whole program into the console everytime :-)
I have installed Python 2.7 and Tcl/Tk 8.5 (remember: it works in the console)
EDIT: So here's my solution:
First, I create a class CSoundPlayer:
from Tkinter import*
import tkSnack
class CSoundPlayer:
def __init__(self, callbackFunction):
self.__activated = False
self.__callbackFunction = callbackFunction
self.__sounds = []
self.__numberOfSounds = 0
self.__root = Tk()
self.__root.title("SoundPlayer")
tkSnack.initializeSnack(self.__root)
def __mainFunction(self):
self.__callbackFunction()
self.__root.after(1, self.__mainFunction)
pass
def activate(self):
self.__activated = True
self.__root.after(1, self.__mainFunction)
self.__root.mainloop()
def loadFile(self, fileName):
if self.__activated:
self.__sounds.append(tkSnack.Sound(load=fileName))
self.__numberOfSounds += 1
# return the index of the new sound
return self.__numberOfSounds - 1
else:
return -1
def play(self, soundIndex):
if self.__activated:
self.__sounds[soundIndex].play()
else:
return -1
Then, the application itself must be implemented in a class thus the main() is defined when handed over to the CSoundPlayer() constructor:
class CApplication:
def __init__(self):
self.__programCounter = -1
self.__SoundPlayer = CSoundPlayer(self.main)
self.__SoundPlayer.activate()
def main(self):
self.__programCounter += 1
if self.__programCounter == 0:
self.__sound1 = self.__SoundPlayer.loadFile("../mysong.mp3")
self.__SoundPlayer.play(self.__sound1)
# here the cyclic code starts:
print self.__programCounter
CApplication()
As you can see, the mainloop() is called not in the constructor but in the activate() method. This is because the CApplication won't ever get the reference to CSoundPlayer object because that stucks in the mainloop.
The code of the class CApplication itself does a lot of overhead. The actual "application code" is placed inside the CApplication.main() - code which shall be executed only once is controlled by means of the program counter.
Now I put it to the next level and place a polling process of the MIDI Device in the CApplication.main(). Thus I will use MIDI commands as trigger for playing sound files. I hope the performance is sufficient for appropriate latency.
Have you any suggestions for optimization?
You must start the event loop. Without the event loop, tkinter has no way to actually draw the window. Remove the while loop and replace it with mainloop:
from Tkinter import *
root = Tk()
root.mainloop()
If you need to do polling (as mentioned in the comments to the question), write a function that polls, and have that function run every periodically with after:
def poll():
<do the polling here>
# in 100ms, call the poll function again
root.after(100, poll)
The reason you don't need mainloop in the console depends on what you mean by "the console". In IDLE, and perhaps some other interactive interpreters, tkinter has a special mode when running interactively that doesn't require you call mainloop. In essence, the mainloop is the console input loop.
I didn't find quite what I was looking for.
I want to obtain the output (stdout) from a python function in real time.
The actual problem is that I want to plot a graph (with cplot from sympy) with a progress bar in my UI. The argument verbose makes cplot output the progress to stdout.
sympy.mpmath.cplot(lambda z: z, real, imag, verbose=True)
The output would be something like:
0 of 71
1 of 71
2 of 71
...
And so on.
I want to capture line by line so I can make a progress bar. (I realize this might not be possible without implementing multithreading). I'm using python2.7 (mainly because I need libraries that aren't in python3)
So, ¿How do I achieve that?
You can capture stdout by monkeypatching sys.stdout. A good way to do it is using a context manager, so that it gets put back when you are done (even if the code raises an exception). If you don't use a context manager, be sure to put the original sys.stdout back using a finally block.
You'll need an object that is file-like, that takes the input and does what you want with it. Subclassing StringIO is a good start. Here's an example of a context manager that captures stdout and stderr and puts them in the result of the bound variable.
class CapturedText(object):
pass
#contextmanager
def captured(disallow_stderr=True):
"""
Context manager to capture the printed output of the code in the with block
Bind the context manager to a variable using `as` and the result will be
in the stdout property.
>>> from tests.helpers import capture
>>> with captured() as c:
... print('hello world!')
...
>>> c.stdout
'hello world!\n'
"""
import sys
stdout = sys.stdout
stderr = sys.stderr
sys.stdout = outfile = StringIO()
sys.stderr = errfile = StringIO()
c = CapturedText()
yield c
c.stdout = outfile.getvalue()
c.stderr = errfile.getvalue()
sys.stdout = stdout
sys.stderr = stderr
if disallow_stderr and c.stderr:
raise Exception("Got stderr output: %s" % c.stderr)
(source)
It works as shown in the docstring. You can replace StringIO() with your own class that writes the progress bar.
Another possibility would be to monkeypatch sympy.mpmath.visualization.print, since cplot uses print to print the output, and it uses from __future__ import print_function.
First, make sure you are using from __future__ import print_function if you aren't using Python 3, as this will otherwise be a SyntaxError.
Then something like
def progressbar_print(*args, **kwargs):
# Take *args and convert it to a progress output
progress(*args)
# If you want to still print the output, do it here
print(*args, **kwargs)
sympy.mpmath.visualization.print = progressbar_print
You might want to monkeypatch it in a custom function that puts it back, as other functions in that module might use print as well. Again, remember to do this using either a context manager or a finally block so that it gets put back even if an exception is raised.
Monkeypatching sys.stdout is definitely the more standard way of doing this, but I like this solution in that it shows that having print as a function can actually be useful.
I want to be able to interrupt a long function with cython, using the usual CTRL+C interrupt command.
My C++ long function is repeatedly called inside a while loop from Cython code, but I want to be able, during the loop, to send an "interrupt" and block the while loop.
The interrupt also should wait the longFunction() to finish, so that no data are lost or kept in unknown status.
This is one of my first implementation, which obviously doesn't work:
computed=0;
print "Computing long function..."
while ( computed==0 ):
try:
computed = self.thisptr.aLongFunction()
except (KeyboardInterrupt, SystemExit):
computed=1
print '\n! Received keyboard interrupt.\n'
break;
(p.s. self.thisptr is the pointer to the current class which implements aLongFunction() )
You should be able to do something like this:
import signal
class Test():
def __init__(self):
self.loop_finished = False
signal.signal(signal.SIGINT, self.abort_loop)
def abort_loop(self, signal, frame):
self.loop_finished = True
def go(self):
while not self.loop_finished:
print "Calculating"
# Do your calculations
# Once calcations are done, set self.loop_finished to True
print "Calculating over"
Test().go()
You can also use additional variables to keep track of whether the computation was manually aborted or not.
I am not a Python C-Api foo master, however this works, but maybe its not the best way:
cdef extern from "Python.h":
int PyErr_CheckSignals()
def test_interrupt():
cdef int i = 0
while i < 9999999999 and not PyErr_CheckSignals():
# do fancy stuff.
i += 1
Of course that is not asynchronous which is maybe possible but I do not know and this breaks the loop for any signal, not just Ctrl+C. Also maybe its better to not check signals every loop iteration, at least if what is done is very cheap.
You could call PyErr_Occurred, etc. if PyErr_CheckSignals did not return 0 and see what kind of Exception got raised to go specifically for KeybordInterrupt or certain kill signals. (Anyway, check the python C-Api for details there...)
If you are calling a cython cdef function inside the while loop you may be able to achieve this also if you add except *:
cdef function(...) except *:
pass
see also http://docs.cython.org/src/userguide/language_basics.html#error-return-values
What is the design pattern behind python like interactive shell. I want to do this for my server but I am ending up with lot of if - then- else pattern.
For example, when I start python interpreter I get something like this
Python 2.6.7 (r267:88850, Feb 2 2012, 23:50:20)
[GCC 4.5.3] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> help
After help the prompt changes to help
Welcome to Python 2.6! This is the online help utility.
If this is your first time using Python, you should definitely check out
the tutorial on the Internet at http://docs.python.org/tutorial/.
Enter the name of any module, keyword, or topic to get help on writing
Python programs and using Python modules. To quit this help utility and
return to the interpreter, just type "quit".
To get a list of available modules, keywords, or topics, type "modules",
"keywords", or "topics". Each module also comes with a one-line summary
of what it does; to list the modules whose summaries contain a given word
such as "spam", type "modules spam".
help>
I think this is some king of read-eval loop design.
for a REPL, you need a context (an object which stores the REPL's state), a command parser (which parses input and produces an AST), and a way to map commands to actions (actions are generally just functions that modifies the context and/or produces side effects).
A simple REPL can be implemented like the following, where context is implemented using a simple dictionary, AST is just the inputted commands split on whitespaces, and a dictionary is used to map commands to actions:
context = {}
commands = {}
def register(func):
""" convenience function to put `func` into commands map """
# in C++, you cannot introspect the function's name so you would
# need to map the function name to function pointers manually
commands[func.__name__] = func
def parse(s):
""" given a command string `s` produce an AST """
# the simplest parser is just splitting the input string,
# but you can also produce use a more complicated grammer
# to produce a more complicated syntax tree
return s.split()
def do(cmd, commands, context):
""" evaluate the AST, producing an output and/or side effect """
# here, we simply use the first item in the list to choose which function to call
# in more complicated ASTs, the type of the root node can be used to pick actions
return commands[cmd[0]](context, cmd)
#register
def assign(ctx, args):
ctx[args[1]] = args[2]
return '%s = %s' % (args[1], args[2])
#register
def printvar(ctx, args):
print ctx[args[1]]
return None
#register
def defun(ctx, args):
body = ' '.join(args[2:])
ctx[args[1]] = compile(body, '', 'exec')
return 'def %s(): %s' % (args[1], body)
#register
def call(ctx, args):
exec ctx[args[1]] in ctx
return None
# more commands here
context['PS1'] = "> "
while True:
# READ
inp = raw_input(context["PS1"])
# EVAL
cmd = parse(inp)
out = do(cmd, commands, context)
# PRINT
if out is not None: print out
# LOOP
A sample session:
> assign d hello
d = hello
> printvar d
hello
> assign PS1 $
PS1 = $
$defun fun print d + 'world'
def fun(): print d + 'world'
$call fun
helloworld
with a little bit more trickery, you could even merge the context and commands dictionary together, allowing the shell's set of commands to be modified in the shell's language.
The name of this design pattern, if it has a name, is Read-Eval-Print Loop design pattern; so yeah, your question sorta answers itself.