I am using various xunit tests for years (starting from cppunit in early 2000s). In all cases it was very easy to set a break point on failure: there was a function that indicated detected failure:
b 'atf::tests::tc::fail(std::string const&)'
command
up 1
end
It seems that gtest is quite different, what is the established practice of doing the same with gtest?
If you need to break at the start of the test to observe something, first get the symbol names present in the executable, and grep for the test name of interest, e.g.:
nm -C myclass_test | grep MyTest0
if you want to break at:
TEST(MainTest, MyTest0) {
EXPECT_EQ(1, 1);
}
Of the results of that grep, the most promising one seemed to be:
0000000000407c64 T MainTest_MyTest0_Test::TestBody()
and so:
gdb myclass_test
and:
b MainTest_MyTest0_Test::TestBody
r
and then this leaves me at the start of the desired test.
Tested with this setup at revision 2.
what is the established practice of doing the same with gtest?
Reading gtest.cc, the closest I see is --gunit_break_on_failure, which should cause the code to execute INT3 trap on x86/Linux, and to call DebugBreak on Windows.
Update: the flag appears to have been renamed to --gtest_break_on_failure in latest public releases.
Related
I'm working on a TCL project, for which version control is based on Git. So, in order to produce good-quality code, I set up put execution of the tests in pre-commit hook.
However, even if they are executed (trace is shown in command-line), and one of the tests is failed, Git performs the commit. So I launched the hook manually to check the error code, and I figured out that it is null, explaining why Git does not stop:
$ .git/hooks/pre-commit
++++ FlattenResult-test PASSED
(...)
==== CheckF69F70 FAILED
==== Content of test case:
(...)
==== CheckF69F70 FAILED
$ echo $?
0
(Launching the tests script with tclsh also results in $? to be 0.)
So my question is about this last line: why is $? equal to 0, when one of the tcl tests is failed? And how can I achieve a simple pre-commit hook that stops on failure?
I read and reread the tcltest documentation, but saw no setting or information about this error code. And I would really like not to have to parse the tcl tests output, to check if ERROR or FAILED is present...
Edit: versions
TCL version : 8.5
tcltest version: 2.3.4
This depends on how you run your test suite. Normally you run a file called tests/all.tcl which may look something like this:
package require Tcl 8.6
package require tcltest 2.5
namespace import tcltest::*
configure -testdir [file dirname [file normalize [info script]]] {*}$argv
runAllTests
That final runAllTests returns a boolean indicating success (0) or failure (1). You can use that to generate an exit code by changing the last line to:
exit [runAllTests]
I use this redefinition in some of my test scripts:
# Exit non-zero if any tests fail.
# tcltest's `cleanupTests` resets the numTests array, so capture it first.
proc cleanupTests {} {
set failed [expr {$::tcltest::numTests(Failed) > 0}]
uplevel 1 ::tcltest::cleanupTests
if {$failed} then {exit 1}
}
After some research, I could make it work, even though several factors were against me:
I have to use an old TCL version (8.5) with tcltest version 2.3.4, in which runAllTests returns nothing;
I forgot to write cleanupTests at the end of test scripts, as the documentation is not really clear about its usage. (It is not clearer now. I just figured out it is needed if you want to get your tests run by runAllTests, which is really not obvious).
And here is my solution, mostly based on Hai's DevBits blog post:
all.tcl
package require tcltest
::tcltest::configure (...)
proc ::tcltest::cleanupTestsHook {} {
variable numTests
set ::exitCode [expr {$numTests(Total) == 0 || $numTests(Failed) > 0}]
}
::tcltest::runAllTests
exit $exitCode
Some thoughts about it:
I added $numTests(Total) == 0 as a failure condition: this means that no tests was found, which is clearly an erroneous condition;
This doesn't catch exceptions in the configuration of the tests, for instance a source command that points to a non-existing file, revealing some failure in tests scaffolding. This would be catched as error in other test framewords (ah, pytest, I miss you!)
How can we list all the functions being called in an application. I tried using GDB but its backtrace list only upto the main function call.
I need deeper list i.e list of all the functions being called by the main function and the function being called from these called functions and so on.
Is there a way to get this in gdb? Or could you give me suggestions on how to get this?
How can we list all the functions being called in an application
For any realistically sized application, this list will have thousands of entries, which will probably make it useless.
You can find out all functions defined (but not necessarily called) in an application with the nm command, e.g.
nm /path/to/a.out | egrep ' [TW] '
You can also use GDB to set a breakpoint on each function:
(gdb) set logging on # collect trace in gdb.txt
(gdb) set confirm off # you wouldn't want to confirm every one of them
(gdb) rbreak . # set a breakpoint on each function
Once you continue, you'll hit a breakpoint for each function called. Use the disable and continue commands to move forward. I don't believe there is an easy way to automate that, unless you want to use Python scripting.
Already mentioned gprof is another good option.
You want a call graph. The tool that you want to use is not gdb, it's gprof. You compile your program with -pg and then run it. When it runs a file gmon.out will be produced. You then process this file with gprof and enjoy the output.
record function-call-history
https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html
This should be a great hardware accelerated possibility if you are one of the few people (2015) with a CPU that supports Intel Processor Tracing (Intel PT, intel_pt in /proc/cpuinfo).
GDB docs claim that it can produce output like:
(gdb) list 1, 10
1 void foo (void)
2 {
3 }
4
5 void bar (void)
6 {
7 ...
8 foo ();
9 ...
10 }
(gdb) record function-call-history /ilc
1 bar inst 1,4 at foo.c:6,8
2 foo inst 5,10 at foo.c:2,3
3 bar inst 11,13 at foo.c:9,10
Before using it you need to run:
start
record btrace
which is where a non capable CPU fails with:
Target does not support branch tracing.
CPU support is further discussed at: How to run record instruction-history and function-call-history in GDB?
Related threads:
how to trace function call in C?
Is there a compiler feature to inject custom function entry and exit code?
For embedded, you also consider JTAG and supporting hardware like ARM's DSTREAM, but x86 support does not seem very good: debugging x86 kernel using a hardware debugger
This question might need clarification to decide between what are currently 2 answers. Depends on what you need:
1) You need to know how many times each function is being called in straight list/graph format of functions matched with # of calls. This could lead to ambiguous/inconclusive results if your code is not procedural (i.e. functions calling other functions in a branch out structure without ambiguity of what is calling what). This is basic gprof functionality which requires recompilation with -pg flag.
2) You need a list of functions in the order in which they were called, this depends on your program which is the best/feasible option:
a) IF your program runs and terminates without runtime errors you can use gprof for this purpose.
b) ELSE option above using dbg with logging and break points is the left over option that I learned upon reading this.
3) You need to know not only the order but, for example, the function arguments for each call as well. My current work is simulations in physics of particle transport, so this would ABSOLUTELY be useful in tracking down where anomalous results are coming from... i.e. when the arguments getting passed around stop making sense. I imagine one way to do this is would be a variation on what Employed Russian did except using the following:
(gdb) info args
Logging the results of this command with every break point (set at every function call) gives the args of the current function.
With gdb, if you can find the most child function, you can list its all ancestors like this:
gdb <your-binary>
(gdb) b theMostChildFunction ## put breakpoint on the desired function
(gdb) r ## run the program
(gdb) bt ## backtrace starting from the breakpoint
Otherwise, on linux, you can use perf tool to trace programs and their function calls. The advantage of this, it is tracing all processes including child processes and also it shows usage percentages of the functions in the program.
You can install perf like this:
sudo apt install linux-tools-generic
sudo apt install linux-cloud-tools-generic
Before using perf you may also need to remove some kernel restrictions temporarily:
sudo sh -c 'echo 0 >/proc/sys/kernel/kptr_restrict'
sudo sh -c 'echo 0 >/proc/sys/kernel/perf_event_paranoid'
sudo sh -c 'echo 0 >/proc/sys/kernel/yama/ptrace_scope'
After this, you can run your program binary with perf like this:
perf record -g -s -a <your-binary-and-its-flags>
Then either you can look the output on terminal like this:
perf report
or on text file like this:
perf report -i perf.data > output.txt
vim output.txt
when you are recording the function calls with perf also you may want to filter kernel calls with --all-user flag:
perf record -g -s -a --all-user <your-binary-and-its-flags>
For further information you can look here: https://perf.wiki.kernel.org/index.php/Tutorial
I'd like to get the number of tests that were run with go test, as kind of a checksum to detect if all the tests are running. Since Go relies on filenames and method names to determine what's a test, it's easy to mistype something, which would mean the test would silently be skipped.
I think that the gotestsum tool is close to what you are looking for.
It is a wrapper around go test that prints formatted test output and a summary of the test run.
Default go test:
go test ./...
? github.com/marco-m/timeit/cmd/sleepit [no test files]
ok github.com/marco-m/timeit/cmd/timeit 0.601s
Default gotestsum:
gotestsum
∅ cmd/sleepit
✓ cmd/timeit (cached)
DONE 11 tests in 0.273s <== see here
Checkout the documentation and the built-in help, it is well-written.
In my experience, gotestsum (and the other tools by the same organization) is good. For me, it is also very important to be able to use the standard Go test package, without other "test frameworks". gotestsum allows me to do so.
On the other hand, to really satisfy your requirement (print the number of declared tests and verify that that number is actually ran), you would need something like TAP, the Test Anything Protocol, which works for any programming language:
1..4 <== see here
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
not ok 4 - Summarized correctly # TODO Not written yet
TAP actually is very nice and simple. I remember there was a Go port, tap-go, but it is now marked as archived.
The go test command has support for the -c flag, described as follows:
-c Compile the test binary to pkg.test but do not run it.
(Where pkg is the last element of the package's import path.)
As far as I understand, generating a binary like this is the way to run it interactively using GDB. However, since the test binary is created by combining the source and test files temporarily in some /tmp/ directory, this is what happens when I run list in gdb:
Loading Go Runtime support.
(gdb) list
42 github.com/<username>/<project>/_test/_testmain.go: No such file or directory.
This means I cannot happily inspect the Go source code in GDB like I'm used to. I know it is possible to force the temporary directory to stay by passing the -work flag to the go test command, but then it is still a huge hassle since the binary is not created in that directory and such. I was wondering if anyone found a clean solution to this problem.
Go 1.5 has been released, and there is still no officially sanctioned Go debugger. I haven't had much success using GDB for effectively debugging Go programs or test binaries. However, I have had success using Delve, a non-official debugger that is still undergoing development: https://github.com/derekparker/delve
To run your test code in the debugger, simply install delve:
go get -u github.com/derekparker/delve/cmd/dlv
... and then start the tests in the debugger from within your workspace:
dlv test
From the debugger prompt, you can single-step, set breakpoints, etc.
Give it a whirl!
Unfortunately, this appears to be a known issue that's not going to be fixed. See this discussion:
https://groups.google.com/forum/#!topic/golang-nuts/nIA09gp3eNU
I've seen two solutions to this problem.
1) create a .gdbinit file with a set substitute-path command to
redirect gdb to the actual location of the source. This file could be
generated by the go tool but you'd risk overwriting someone's custom
.gdbinit file and would tie the go tool to gdb which seems like a bad
idea.
2) Replace the source file paths in the executable (which are pointing
to /tmp/...) with the location they reside on disk. This is
straightforward if the real path is shorter then the /tmp/... path.
This would likely require additional support from the compiler /
linker to make this solution more generic.
It spawned this issue on the Go Google Code issue tracker, to which the decision ended up being:
https://code.google.com/p/go/issues/detail?id=2881
This is annoying, but it is the least of many annoying possibilities.
As a rule, the go tool should not be scribbling in the source
directories, which might not even be writable, and it shouldn't be
leaving files elsewhere after it exits. There is next to nothing
interesting in _testmain.go. People testing with gdb can break on
testing.Main instead.
Russ
Status: Unfortunate
So, in short, it sucks, and while you can work around it and GDB a test executable, the development team is unlikely to make it as easy as it could be for you.
I'm still new to the golang game but for what it's worth basic debugging seems to work.
The list command you're trying to work can be used so long as you're already at a breakpoint somewhere in your code. For example:
(gdb) b aws.go:54
Breakpoint 1 at 0x61841: file /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.go, line 54.
(gdb) r
Starting program: /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.test
[snip: some arnings about BinaryCache]
Breakpoint 1, github.com/stellar/deliverator/aws.imageIsNewer (latest=0xc2081fe2d0, ami=0xc2081fe3c0, ~r2=false)
at /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.go:54
54 layout := "2006-01-02T15:04:05.000Z"
(gdb) list
49 func imageIsNewer(latest *ec2.Image, ami *ec2.Image) bool {
50 if latest == nil {
51 return true
52 }
53
54 layout := "2006-01-02T15:04:05.000Z"
55
56 amiCreationTime, amiErr := time.Parse(layout, *ami.CreationDate)
57 if amiErr != nil {
58 panic(amiErr)
This is just after running the following in the aws subdir of my project:
go test -c
gdb aws.test
As an additional caveat, it does seem very selective about where breakpoints can be placed. Seems like it has to be an expression but that conclusion is only via experimentation.
If you're willing to use tools besides GDB, check out godebug. To use it, first install with:
go get github.com/mailgun/godebug
Next, insert a breakpoint somewhere by adding the following statement to your code:
_ = "breakpoint"
Now run your tests with the godebug test command.
godebug test
It supports many of the parameters from the go test command.
-test.bench string
regular expression per path component to select benchmarks to run
-test.benchmem
print memory allocations for benchmarks
-test.benchtime duration
approximate run time for each benchmark (default 1s)
-test.blockprofile string
write a goroutine blocking profile to the named file after execution
-test.blockprofilerate int
if >= 0, calls runtime.SetBlockProfileRate() (default 1)
-test.count n
run tests and benchmarks n times (default 1)
-test.coverprofile string
write a coverage profile to the named file after execution
-test.cpu string
comma-separated list of number of CPUs to use for each test
-test.cpuprofile string
write a cpu profile to the named file during execution
-test.memprofile string
write a memory profile to the named file after execution
-test.memprofilerate int
if >=0, sets runtime.MemProfileRate
-test.outputdir string
directory in which to write profiles
-test.parallel int
maximum test parallelism (default 4)
-test.run string
regular expression to select tests and examples to run
-test.short
run smaller test suite to save time
-test.timeout duration
if positive, sets an aggregate time limit for all tests
-test.trace string
write an execution trace to the named file after execution
-test.v
verbose: print additional output
How can we list all the functions being called in an application. I tried using GDB but its backtrace list only upto the main function call.
I need deeper list i.e list of all the functions being called by the main function and the function being called from these called functions and so on.
Is there a way to get this in gdb? Or could you give me suggestions on how to get this?
How can we list all the functions being called in an application
For any realistically sized application, this list will have thousands of entries, which will probably make it useless.
You can find out all functions defined (but not necessarily called) in an application with the nm command, e.g.
nm /path/to/a.out | egrep ' [TW] '
You can also use GDB to set a breakpoint on each function:
(gdb) set logging on # collect trace in gdb.txt
(gdb) set confirm off # you wouldn't want to confirm every one of them
(gdb) rbreak . # set a breakpoint on each function
Once you continue, you'll hit a breakpoint for each function called. Use the disable and continue commands to move forward. I don't believe there is an easy way to automate that, unless you want to use Python scripting.
Already mentioned gprof is another good option.
You want a call graph. The tool that you want to use is not gdb, it's gprof. You compile your program with -pg and then run it. When it runs a file gmon.out will be produced. You then process this file with gprof and enjoy the output.
record function-call-history
https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html
This should be a great hardware accelerated possibility if you are one of the few people (2015) with a CPU that supports Intel Processor Tracing (Intel PT, intel_pt in /proc/cpuinfo).
GDB docs claim that it can produce output like:
(gdb) list 1, 10
1 void foo (void)
2 {
3 }
4
5 void bar (void)
6 {
7 ...
8 foo ();
9 ...
10 }
(gdb) record function-call-history /ilc
1 bar inst 1,4 at foo.c:6,8
2 foo inst 5,10 at foo.c:2,3
3 bar inst 11,13 at foo.c:9,10
Before using it you need to run:
start
record btrace
which is where a non capable CPU fails with:
Target does not support branch tracing.
CPU support is further discussed at: How to run record instruction-history and function-call-history in GDB?
Related threads:
how to trace function call in C?
Is there a compiler feature to inject custom function entry and exit code?
For embedded, you also consider JTAG and supporting hardware like ARM's DSTREAM, but x86 support does not seem very good: debugging x86 kernel using a hardware debugger
This question might need clarification to decide between what are currently 2 answers. Depends on what you need:
1) You need to know how many times each function is being called in straight list/graph format of functions matched with # of calls. This could lead to ambiguous/inconclusive results if your code is not procedural (i.e. functions calling other functions in a branch out structure without ambiguity of what is calling what). This is basic gprof functionality which requires recompilation with -pg flag.
2) You need a list of functions in the order in which they were called, this depends on your program which is the best/feasible option:
a) IF your program runs and terminates without runtime errors you can use gprof for this purpose.
b) ELSE option above using dbg with logging and break points is the left over option that I learned upon reading this.
3) You need to know not only the order but, for example, the function arguments for each call as well. My current work is simulations in physics of particle transport, so this would ABSOLUTELY be useful in tracking down where anomalous results are coming from... i.e. when the arguments getting passed around stop making sense. I imagine one way to do this is would be a variation on what Employed Russian did except using the following:
(gdb) info args
Logging the results of this command with every break point (set at every function call) gives the args of the current function.
With gdb, if you can find the most child function, you can list its all ancestors like this:
gdb <your-binary>
(gdb) b theMostChildFunction ## put breakpoint on the desired function
(gdb) r ## run the program
(gdb) bt ## backtrace starting from the breakpoint
Otherwise, on linux, you can use perf tool to trace programs and their function calls. The advantage of this, it is tracing all processes including child processes and also it shows usage percentages of the functions in the program.
You can install perf like this:
sudo apt install linux-tools-generic
sudo apt install linux-cloud-tools-generic
Before using perf you may also need to remove some kernel restrictions temporarily:
sudo sh -c 'echo 0 >/proc/sys/kernel/kptr_restrict'
sudo sh -c 'echo 0 >/proc/sys/kernel/perf_event_paranoid'
sudo sh -c 'echo 0 >/proc/sys/kernel/yama/ptrace_scope'
After this, you can run your program binary with perf like this:
perf record -g -s -a <your-binary-and-its-flags>
Then either you can look the output on terminal like this:
perf report
or on text file like this:
perf report -i perf.data > output.txt
vim output.txt
when you are recording the function calls with perf also you may want to filter kernel calls with --all-user flag:
perf record -g -s -a --all-user <your-binary-and-its-flags>
For further information you can look here: https://perf.wiki.kernel.org/index.php/Tutorial