Related
How can we list all the functions being called in an application. I tried using GDB but its backtrace list only upto the main function call.
I need deeper list i.e list of all the functions being called by the main function and the function being called from these called functions and so on.
Is there a way to get this in gdb? Or could you give me suggestions on how to get this?
How can we list all the functions being called in an application
For any realistically sized application, this list will have thousands of entries, which will probably make it useless.
You can find out all functions defined (but not necessarily called) in an application with the nm command, e.g.
nm /path/to/a.out | egrep ' [TW] '
You can also use GDB to set a breakpoint on each function:
(gdb) set logging on # collect trace in gdb.txt
(gdb) set confirm off # you wouldn't want to confirm every one of them
(gdb) rbreak . # set a breakpoint on each function
Once you continue, you'll hit a breakpoint for each function called. Use the disable and continue commands to move forward. I don't believe there is an easy way to automate that, unless you want to use Python scripting.
Already mentioned gprof is another good option.
You want a call graph. The tool that you want to use is not gdb, it's gprof. You compile your program with -pg and then run it. When it runs a file gmon.out will be produced. You then process this file with gprof and enjoy the output.
record function-call-history
https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html
This should be a great hardware accelerated possibility if you are one of the few people (2015) with a CPU that supports Intel Processor Tracing (Intel PT, intel_pt in /proc/cpuinfo).
GDB docs claim that it can produce output like:
(gdb) list 1, 10
1 void foo (void)
2 {
3 }
4
5 void bar (void)
6 {
7 ...
8 foo ();
9 ...
10 }
(gdb) record function-call-history /ilc
1 bar inst 1,4 at foo.c:6,8
2 foo inst 5,10 at foo.c:2,3
3 bar inst 11,13 at foo.c:9,10
Before using it you need to run:
start
record btrace
which is where a non capable CPU fails with:
Target does not support branch tracing.
CPU support is further discussed at: How to run record instruction-history and function-call-history in GDB?
Related threads:
how to trace function call in C?
Is there a compiler feature to inject custom function entry and exit code?
For embedded, you also consider JTAG and supporting hardware like ARM's DSTREAM, but x86 support does not seem very good: debugging x86 kernel using a hardware debugger
This question might need clarification to decide between what are currently 2 answers. Depends on what you need:
1) You need to know how many times each function is being called in straight list/graph format of functions matched with # of calls. This could lead to ambiguous/inconclusive results if your code is not procedural (i.e. functions calling other functions in a branch out structure without ambiguity of what is calling what). This is basic gprof functionality which requires recompilation with -pg flag.
2) You need a list of functions in the order in which they were called, this depends on your program which is the best/feasible option:
a) IF your program runs and terminates without runtime errors you can use gprof for this purpose.
b) ELSE option above using dbg with logging and break points is the left over option that I learned upon reading this.
3) You need to know not only the order but, for example, the function arguments for each call as well. My current work is simulations in physics of particle transport, so this would ABSOLUTELY be useful in tracking down where anomalous results are coming from... i.e. when the arguments getting passed around stop making sense. I imagine one way to do this is would be a variation on what Employed Russian did except using the following:
(gdb) info args
Logging the results of this command with every break point (set at every function call) gives the args of the current function.
With gdb, if you can find the most child function, you can list its all ancestors like this:
gdb <your-binary>
(gdb) b theMostChildFunction ## put breakpoint on the desired function
(gdb) r ## run the program
(gdb) bt ## backtrace starting from the breakpoint
Otherwise, on linux, you can use perf tool to trace programs and their function calls. The advantage of this, it is tracing all processes including child processes and also it shows usage percentages of the functions in the program.
You can install perf like this:
sudo apt install linux-tools-generic
sudo apt install linux-cloud-tools-generic
Before using perf you may also need to remove some kernel restrictions temporarily:
sudo sh -c 'echo 0 >/proc/sys/kernel/kptr_restrict'
sudo sh -c 'echo 0 >/proc/sys/kernel/perf_event_paranoid'
sudo sh -c 'echo 0 >/proc/sys/kernel/yama/ptrace_scope'
After this, you can run your program binary with perf like this:
perf record -g -s -a <your-binary-and-its-flags>
Then either you can look the output on terminal like this:
perf report
or on text file like this:
perf report -i perf.data > output.txt
vim output.txt
when you are recording the function calls with perf also you may want to filter kernel calls with --all-user flag:
perf record -g -s -a --all-user <your-binary-and-its-flags>
For further information you can look here: https://perf.wiki.kernel.org/index.php/Tutorial
As part of a Compiler Principles course I'm taking in my university, we're writing a compiler that's implemented in OCaml, which compiles Scheme code into CISC-like assembly (which is just C macros).
the basic operation of the compiler is such:
Read a *.scm file and convert it to an OCaml string.
Parse the string and perform various analyses.
Run a code generator on the AST output from the semantic analyzer, that outputs text into a *.c file.
Compile that file with GCC and run it in the terminal.
Well, all is good and well, except for this: I'm trying to read an input file, that's around 4000 lines long, and is basically one huge expressions that's a mix of Scheme if & and.
I'm executing the compiler via utop. When I try to read the input file, I immediately get a stack overflow error message. It is my initial guess that the file is just to large for OCaml to handle, but I wasn't able to find any documentation that would support this theory.
Any suggestions?
The maximum string length is given by Sys.max_string_length. For a 32-bit system, it's quite short: 16777211. For a 64-bit system, it's 144115188075855863.
Unless you're using a 32-bit system, and your 4000-line file is over 16MB, I don't think you're hitting the string length limit.
A stack overflow is not what you'd expect to see when a string is too long.
It's more likely that you have infinite recursion, or possibly just a very deeply nested computation.
Well, it turns out that the limitation was the amount of maximum ram the OCaml is configured to use.
I ran the following command in the terminal in order to increase the quota:
export OCAMLRUNPARAM="l=5555555555"
This worked like a charm - I managed to read and compile the input file almost instantaneously.
For reference purposes, this is the code that reads the file:
let file_to_string input_file =
let in_channel = open_in input_file in
let rec run () =
try
let ch = input_char in_channel in ch :: (run ())
with End_of_file ->
( close_in in_channel;
[] )
in list_to_string (run ());;
where list_to_string is:
let list_to_string s =
let rec loop s n =
match s with
| [] -> String.make n '?'
| car :: cdr ->
let result = loop cdr (n + 1) in
String.set result n car;
result
in
loop s 0;;
funny thing is - I wrote file_to_string in tail recursion. This prevented the stack overflow, but for some reason went into an infinite loop. Oh, well...
How can we list all the functions being called in an application. I tried using GDB but its backtrace list only upto the main function call.
I need deeper list i.e list of all the functions being called by the main function and the function being called from these called functions and so on.
Is there a way to get this in gdb? Or could you give me suggestions on how to get this?
How can we list all the functions being called in an application
For any realistically sized application, this list will have thousands of entries, which will probably make it useless.
You can find out all functions defined (but not necessarily called) in an application with the nm command, e.g.
nm /path/to/a.out | egrep ' [TW] '
You can also use GDB to set a breakpoint on each function:
(gdb) set logging on # collect trace in gdb.txt
(gdb) set confirm off # you wouldn't want to confirm every one of them
(gdb) rbreak . # set a breakpoint on each function
Once you continue, you'll hit a breakpoint for each function called. Use the disable and continue commands to move forward. I don't believe there is an easy way to automate that, unless you want to use Python scripting.
Already mentioned gprof is another good option.
You want a call graph. The tool that you want to use is not gdb, it's gprof. You compile your program with -pg and then run it. When it runs a file gmon.out will be produced. You then process this file with gprof and enjoy the output.
record function-call-history
https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html
This should be a great hardware accelerated possibility if you are one of the few people (2015) with a CPU that supports Intel Processor Tracing (Intel PT, intel_pt in /proc/cpuinfo).
GDB docs claim that it can produce output like:
(gdb) list 1, 10
1 void foo (void)
2 {
3 }
4
5 void bar (void)
6 {
7 ...
8 foo ();
9 ...
10 }
(gdb) record function-call-history /ilc
1 bar inst 1,4 at foo.c:6,8
2 foo inst 5,10 at foo.c:2,3
3 bar inst 11,13 at foo.c:9,10
Before using it you need to run:
start
record btrace
which is where a non capable CPU fails with:
Target does not support branch tracing.
CPU support is further discussed at: How to run record instruction-history and function-call-history in GDB?
Related threads:
how to trace function call in C?
Is there a compiler feature to inject custom function entry and exit code?
For embedded, you also consider JTAG and supporting hardware like ARM's DSTREAM, but x86 support does not seem very good: debugging x86 kernel using a hardware debugger
This question might need clarification to decide between what are currently 2 answers. Depends on what you need:
1) You need to know how many times each function is being called in straight list/graph format of functions matched with # of calls. This could lead to ambiguous/inconclusive results if your code is not procedural (i.e. functions calling other functions in a branch out structure without ambiguity of what is calling what). This is basic gprof functionality which requires recompilation with -pg flag.
2) You need a list of functions in the order in which they were called, this depends on your program which is the best/feasible option:
a) IF your program runs and terminates without runtime errors you can use gprof for this purpose.
b) ELSE option above using dbg with logging and break points is the left over option that I learned upon reading this.
3) You need to know not only the order but, for example, the function arguments for each call as well. My current work is simulations in physics of particle transport, so this would ABSOLUTELY be useful in tracking down where anomalous results are coming from... i.e. when the arguments getting passed around stop making sense. I imagine one way to do this is would be a variation on what Employed Russian did except using the following:
(gdb) info args
Logging the results of this command with every break point (set at every function call) gives the args of the current function.
With gdb, if you can find the most child function, you can list its all ancestors like this:
gdb <your-binary>
(gdb) b theMostChildFunction ## put breakpoint on the desired function
(gdb) r ## run the program
(gdb) bt ## backtrace starting from the breakpoint
Otherwise, on linux, you can use perf tool to trace programs and their function calls. The advantage of this, it is tracing all processes including child processes and also it shows usage percentages of the functions in the program.
You can install perf like this:
sudo apt install linux-tools-generic
sudo apt install linux-cloud-tools-generic
Before using perf you may also need to remove some kernel restrictions temporarily:
sudo sh -c 'echo 0 >/proc/sys/kernel/kptr_restrict'
sudo sh -c 'echo 0 >/proc/sys/kernel/perf_event_paranoid'
sudo sh -c 'echo 0 >/proc/sys/kernel/yama/ptrace_scope'
After this, you can run your program binary with perf like this:
perf record -g -s -a <your-binary-and-its-flags>
Then either you can look the output on terminal like this:
perf report
or on text file like this:
perf report -i perf.data > output.txt
vim output.txt
when you are recording the function calls with perf also you may want to filter kernel calls with --all-user flag:
perf record -g -s -a --all-user <your-binary-and-its-flags>
For further information you can look here: https://perf.wiki.kernel.org/index.php/Tutorial
I'm an amateur C++ programmer trying to learn about basic shell scripting. I have a complex C++ program that currently reads in different parameter values from Parameters.h and then executes one or more simulations with each parameter value sequentially. These simulations take a long time to run. Since I have a cluster available, I'd like to effectively parallelize this job, running the simulations for each parameter value on a separate processor. I'm assuming it's easier to learn shell scripting techniques for this purpose than OpenMPI. My cluster runs on the LSF platform.
How can I write my input parameters in Bash so that they are distributed among multiple processors, each executing the program with that value? I'd like to avoid interactive submission. Ideally, I'd have the inputs in a text file that Bash reads, and I'd be passing two parameters to each job: an actual parameter value and a parameter ID.
Thanks in advance for any leads and suggestions.
my solution
GNU Parallel does look slick, but I ended up (with the help of an IT admin) writing a simple bash script that echos to screen three inputs (a treatment identifier, treatment/parameter value, and a simulation identifier):
#!/bin/bash
j=1
for treatment in cat treatments.txt; do
for experiment in cat simulations.txt; do
bsub -oo tr_${j}_sim_${experiment}_screen -eo tr_${j}_sim_${experiment}_err -q short_serial "echo \"$j $treatment $experiment\" | ./a.out"
done
let j=$j+1
done
The file treatments.txt contains a list of the values I'd like to vary, simulations.txt contains a list of all the simulation identifiers I'd like to run (currently just 1,...,s, where s is the total number of simulations I want for each treatment), and the treatments are indexed 1...j.
Maybe check out: http://www.gnu.org/software/parallel/
edit:
Or, check out the -P argument to xargs, example:
time echo {1..5} | xargs -n 1 -P 5 sleep
Say you want to run the program simulate with inputs foo, bar, baz and quux in parallel, then the simplest way is:
inputs="foo bar baz quux"
# Launch processes in the background with &
children=""
for x in $inputs; do
simulate "$x" > "$x.output" &
$children = "$children $!"
done
# Wait for each to finish
for $pid in $children; do
wait $pid
done
for x in $inputs; do
echo "simulate '$x' gave:"
cat "$x.output"
rm -f "$x.output"
done
The problem is that all simulations are launched at the same time, so if your number of inputs is much larger than your number of CPUs/cores, they may swamp the system.
My best stab at this is you background multiple instances of your program and let the OS's scheduler take over to put them on different processors. AFAIK there is no way in any shell to specify which processor a given process should run on.
Something to the effect of:
#!/bin/sh
for arg in foo bar baz; do
./your_program "$arg" &
done
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
This question's answers are a community effort. Edit existing answers to improve this post. It is not currently accepting new answers or interactions.
I have come to realize that Windbg is a very powerful debugger for the Windows platform & I learn something new about it once in a while. Can fellow Windbg users share some of their mad skills?
ps: I am not looking for a nifty command, those can be found in the documentation. How about sharing tips on doing something that one couldn't otherwise imagine could be done with windbg? e.g. Some way to generate statistics about memory allocations when a process is run under windbg.
My favorite is the command .cmdtree <file> (undocumented, but referenced in previous release notes). This can assist in bringing up another window (that can be docked) to display helpful or commonly used commands. This can help make the user much more productive using the tool.
Initially talked about here, with an example for the <file> parameter:
http://blogs.msdn.com/debuggingtoolbox/archive/2008/09/17/special-command-execute-commands-from-a-customized-user-interface-with-cmdtree.aspx
Example:
alt text http://blogs.msdn.com/photos/debuggingtoolbox/images/8954736/original.aspx
To investigate a memory leak in a crash dump (since I prefer by far UMDH for live processes).
The strategy is that objects of the same type are all allocated with the same size.
Feed the !heap -h 0 command to WinDbg's command line version cdb.exe (for greater speed) to get all heap allocations:
"C:\Program Files\Debugging Tools for Windows\cdb.exe" -c "!heap -h 0;q" -z [DumpPath] > DumpHeapEntries.log
Use Cygwin to grep the list of allocations, grouping them by size:
grep "busy ([[:alnum:]]\+)" DumpHeapEntries.log \
| gawk '{ str = $8; gsub(/\(|\)/, "", str); print "0x" str " 0x" $4 }' \
| sort \
| uniq -c \
| gawk '{ printf "%10.2f %10d %10d ( %s = %d )\n", $1*strtonum($3)/1024, $1, strtonum($3), $2, strtonum($2) }' \
| sort > DumpHeapEntriesStats.log
You get a table that looks like this, for example, telling us that 25529270 allocations of 0x24 bytes take nearly 1.2 GB of memory.
8489.52 707 12296 ( 0x3000 = 12288 )
11894.28 5924 2056 ( 0x800 = 2048 )
13222.66 846250 16 ( 0x2 = 2 )
14120.41 602471 24 ( 0x2 = 2 )
31539.30 2018515 16 ( 0x1 = 1 )
38902.01 1659819 24 ( 0x1 = 1 )
40856.38 817 51208 ( 0xc800 = 51200 )
1196684.53 25529270 48 ( 0x24 = 36 )
Then if your objects have vtables, just use the dps command to seek some of the 0x24 bytes heap allocations in DumpHeapEntries.log to know the type of the objects that are taking all the memory.
0:075> dps 3be7f7e8
3be7f7e8 00020006
3be7f7ec 090c01e7
3be7f7f0 0b40fe94 SomeDll!SomeType::`vftable'
3be7f7f4 00000000
3be7f7f8 00000000
It's cheesy but it works :)
The following command comes very handy when looking on the stack for C++ objects with vtables, especially when working with release builds when quite a few things get optimized away.
dpp esp Range
Being able to load an arbitrary PE file as dump is neat:
windbg -z mylib.dll
Query GetLastError() with:
!gle
This helps to decode common error codes:
!error error_number
Almost 60% of the commands I use everyday..
dv /i /t
?? this
kM (kinda undocumented) generates links to frames
.frame x
!analyze -v
!lmi
~
Explanation
dv /i /t [doc]
dv - display names and values of local variables in the current scope
/i - specify the kind of variable: local, global, parameter, function, or unknown
/t - display data type of variables
?? this [doc]
?? - evaluate C++ expression
this - C++ this pointer
kM [doc]
k - display stack back trace
M - DML mode. Frame numbers are hyperlinks to the particular frame. For more info about kM refer to http://windbg.info/doc/1-common-cmds.html
.frame x [doc]
Switch to frame number x. 0 being the frame at top of stack, 1 being frame 1 below the 0th frame, and so on.
To display local variables from another frame on the stack, first switch to that frame - .frame x, then use dv /i /t. By default d will show info from top frame.
!analyze -v [doc1] [doc2 - Using the !analyze Extension]
!analyze - analyze extension. Display information about the current exception or bug check. Note that to run an extension we prefix !.
-v - verbose output
!lmi [doc]
!lmi - lmi extension. Display detailed information about a module.
~ [doc]
~ - Displays status for the specified thread or for all threads in the current process.
The "tip" I use most often is one that will save you from having to touch that pesky mouse so often: Alt + 1
Alt + 1 will place focus into the command window so that you can actually type a command and so that up-arrow actually scrolls through command history. However, it doesn't work if your focus is already in the scrollable command history.
Peeve: why the heck are key presses ignored while the focus is in a source window? It's not like you can edit the source code from inside WinDbg. Alt + 1 to the rescue.
One word (well, OK, three) : DML, i.e. Debugger Markup Language.
This is a fairly recent addition to WinDbg, and it's not documented in the help file. There is however some documentation in "dml.doc" in the installation directory for the Debugging Tools for Windows.
Basically, this is an HTML-like syntax you can add to your debugger scripts for formatting and, more importantly, linking. You can use links to call other scripts, or even the same script.
My day-to-day work involves maintenance on a meta-modeler that provides generic objects and relationship between objects for a large piece of C++ software. At first, to ease debugging, I had written a simple dump script that extracts relevant information from these objects.
Now, with DML, I've been able to add links to the output, allowing the same script to be called again on related objects. This allows for much faster exploration of a model.
Here's a simplified example. Assume the object under introspection has a relationship called "reference" to another object.
r #$t0 = $arg1 $$ arg1 is the address of an object to examine
$$ dump some information from $t0
$$ allow the user to examine our reference
aS /x myref ##(&((<C++ type of the reference>*)#$t0)->reference )
.block { .printf /D "<link cmd=\"$$>a< <full path to this script> ${myref}\">dump Ref</link> " }
Obviously, this a pretty canned example, but this stuff is really invaluable for me. Instead of hunting around in very complex objects for the right data members (which usually took up to a minute and various casting and dereferencing trickery), everything is automated in one click!
.prefer_dml 1
This modifies many of the built in commands (for example, lm) to display DML output which allows you to click links instead of running commands. Pretty handy...
.reload /f /o file.dll (the /o will overwrite the current copy of the symbol you have)
.enable_unicode 1 //Switches the debugger to default to Unicode for strings since all the Windows components use Unicode internally, this is pretty handy.
.ignore_missing_pages 1 //If you do a lot of kernel dump analysis, you will see a lot of errors regarding memory being paged out. This command will tell the debugger to stop throwing this warning.
alias alias alias...
Save yourself some time in the debugger. Here are some of mine:
aS !p !process;
aS !t !thread;
aS .f .frame;
aS .p .process /p /r
aS .t .thread /p /r
aS dv dv /V /i /t //make dv do your favorite options by default
aS f !process 0 0 //f for find, e.g. f explorer.exe
Another answer mentioned the command window and Alt + 1 to focus on the command input window. Does anyone find it difficult to scroll the command output window without using the mouse?
Well, I have recently used AutoHotkey to scroll the command output window using keyboard and without leaving the command input window.
; WM_VSCROLL = 0x115 (277)
ScrollUp(control="")
{
SendMessage, 277, 0, 0, %control%, A
}
ScrollDown(control="")
{
SendMessage, 277, 1, 0, %control%, A
}
ScrollPageUp(control="")
{
SendMessage, 277, 2, 0, %control%, A
}
ScrollPageDown(control="")
{
SendMessage, 277, 3, 0, %control%, A
}
ScrollToTop(control="")
{
SendMessage, 277, 6, 0, %control%, A
}
ScrollToBottom(control="")
{
SendMessage, 277, 7, 0, %control%, A
}
#IfWinActive, ahk_class WinDbgFrameClass
; For WinDbg, when the child window is attached to the main window
!UP::ScrollUp("RichEdit50W1")
^k::ScrollUp("RichEdit50W1")
!DOWN::ScrollDown("RichEdit50W1")
^j::ScrollDown("RichEdit50W1")
!PGDN::ScrollPageDown("RichEdit50W1")
!PGUP::ScrollPageUp("RichEdit50W1")
!HOME::ScrollToTop("RichEdit50W1")
!END::ScrollToBottom("RichEdit50W1")
#IfWinActive, ahk_class WinBaseClass
; Also for WinDbg, when the child window is a separate window
!UP::ScrollUp("RichEdit50W1")
!DOWN::ScrollDown("RichEdit50W1")
!PGDN::ScrollPageDown("RichEdit50W1")
!PGUP::ScrollPageUp("RichEdit50W1")
!HOME::ScrollToTop("RichEdit50W1")
!END::ScrollToBottom("RichEdit50W1")
After this script is run, you can use Alt + up/down to scroll one line of the command output window, Alt + PgDn/PgUp to scroll one screen.
Note: it seems different versions of WinDbg will have different class names for the window and controls, so you might want to use the window spy tool provided by AutoHotkey to find the actual class names first.
Script to load SOS based on the .NET framework version (v2.0 / v4.0):
!for_each_module .if(($sicmp( "##ModuleName" , "mscorwks") = 0) )
{.loadby sos mscorwks} .elsif ($sicmp( "##ModuleName" , "clr") = 0)
{.loadby sos clr}
I like to use advanced breakpoint commands, such as using breakpoints to create new one-shot breakpoints.
Do not use WinDbg's .heap -stat command. It will sometimes give you incorrect output. Instead, use DebugDiags memory reporting.
Having the correct numbers, you can then use WinDbg's .heap -flt ... command.
For command & straightforward (static or automatable) routines where the debugger is used, it is very cool to be able to put all the debugger commands to run through in a text command file and run that as input through kd.exe or cdb.exe, callable via a batch script, etc.
Run that whenever you need to do this same old routine, without having to fire up WinDbg and do things manually. Too bad this doesn't work when you aren't sure what you are looking for, or some command parameters need manual analysis to find/get.
Platform-independent dump string for managed code which will work for x86/x64:
j $ptrsize = 8 'aS !ds .printf "%mu \n", c+';'aS !ds .printf "%mu \n", 10+'
Here is a sample usage:
0:000> !ds 00000000023620b8
MaxConcurrentInstances