I have a project in stack with tests that I run with stack test.
A test fails and the output says
...
To rerun use: --match "/foo/must be 4/"
....
But how am I supposed to apply that?
stack test --match "/foo/must be 4/" yields Invalid option `--match`.
--match "/foo/must be 4/ is supposed to be given as an argument to the compiled executable program running the test rather than Stack. Pass it to the executable by doing stack test --test-arguments '--match "/foo/must be 4/"' or stack test --ta '--match "/foo/must be 4/"'
Related
How can we list all the functions being called in an application. I tried using GDB but its backtrace list only upto the main function call.
I need deeper list i.e list of all the functions being called by the main function and the function being called from these called functions and so on.
Is there a way to get this in gdb? Or could you give me suggestions on how to get this?
How can we list all the functions being called in an application
For any realistically sized application, this list will have thousands of entries, which will probably make it useless.
You can find out all functions defined (but not necessarily called) in an application with the nm command, e.g.
nm /path/to/a.out | egrep ' [TW] '
You can also use GDB to set a breakpoint on each function:
(gdb) set logging on # collect trace in gdb.txt
(gdb) set confirm off # you wouldn't want to confirm every one of them
(gdb) rbreak . # set a breakpoint on each function
Once you continue, you'll hit a breakpoint for each function called. Use the disable and continue commands to move forward. I don't believe there is an easy way to automate that, unless you want to use Python scripting.
Already mentioned gprof is another good option.
You want a call graph. The tool that you want to use is not gdb, it's gprof. You compile your program with -pg and then run it. When it runs a file gmon.out will be produced. You then process this file with gprof and enjoy the output.
record function-call-history
https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html
This should be a great hardware accelerated possibility if you are one of the few people (2015) with a CPU that supports Intel Processor Tracing (Intel PT, intel_pt in /proc/cpuinfo).
GDB docs claim that it can produce output like:
(gdb) list 1, 10
1 void foo (void)
2 {
3 }
4
5 void bar (void)
6 {
7 ...
8 foo ();
9 ...
10 }
(gdb) record function-call-history /ilc
1 bar inst 1,4 at foo.c:6,8
2 foo inst 5,10 at foo.c:2,3
3 bar inst 11,13 at foo.c:9,10
Before using it you need to run:
start
record btrace
which is where a non capable CPU fails with:
Target does not support branch tracing.
CPU support is further discussed at: How to run record instruction-history and function-call-history in GDB?
Related threads:
how to trace function call in C?
Is there a compiler feature to inject custom function entry and exit code?
For embedded, you also consider JTAG and supporting hardware like ARM's DSTREAM, but x86 support does not seem very good: debugging x86 kernel using a hardware debugger
This question might need clarification to decide between what are currently 2 answers. Depends on what you need:
1) You need to know how many times each function is being called in straight list/graph format of functions matched with # of calls. This could lead to ambiguous/inconclusive results if your code is not procedural (i.e. functions calling other functions in a branch out structure without ambiguity of what is calling what). This is basic gprof functionality which requires recompilation with -pg flag.
2) You need a list of functions in the order in which they were called, this depends on your program which is the best/feasible option:
a) IF your program runs and terminates without runtime errors you can use gprof for this purpose.
b) ELSE option above using dbg with logging and break points is the left over option that I learned upon reading this.
3) You need to know not only the order but, for example, the function arguments for each call as well. My current work is simulations in physics of particle transport, so this would ABSOLUTELY be useful in tracking down where anomalous results are coming from... i.e. when the arguments getting passed around stop making sense. I imagine one way to do this is would be a variation on what Employed Russian did except using the following:
(gdb) info args
Logging the results of this command with every break point (set at every function call) gives the args of the current function.
With gdb, if you can find the most child function, you can list its all ancestors like this:
gdb <your-binary>
(gdb) b theMostChildFunction ## put breakpoint on the desired function
(gdb) r ## run the program
(gdb) bt ## backtrace starting from the breakpoint
Otherwise, on linux, you can use perf tool to trace programs and their function calls. The advantage of this, it is tracing all processes including child processes and also it shows usage percentages of the functions in the program.
You can install perf like this:
sudo apt install linux-tools-generic
sudo apt install linux-cloud-tools-generic
Before using perf you may also need to remove some kernel restrictions temporarily:
sudo sh -c 'echo 0 >/proc/sys/kernel/kptr_restrict'
sudo sh -c 'echo 0 >/proc/sys/kernel/perf_event_paranoid'
sudo sh -c 'echo 0 >/proc/sys/kernel/yama/ptrace_scope'
After this, you can run your program binary with perf like this:
perf record -g -s -a <your-binary-and-its-flags>
Then either you can look the output on terminal like this:
perf report
or on text file like this:
perf report -i perf.data > output.txt
vim output.txt
when you are recording the function calls with perf also you may want to filter kernel calls with --all-user flag:
perf record -g -s -a --all-user <your-binary-and-its-flags>
For further information you can look here: https://perf.wiki.kernel.org/index.php/Tutorial
Is there a way to rerun entire Test Suite if a single Particular test case fails .
So for example , a Robot Code which contain a test case which will check the cookie value , if the cookie is of a particular pattern will continue execution of rest of code , if it fails it should rerun the entire Robot Code / Test Suite and repeat this 3 times , if the cookie value is not same for three runs , let it fail the test suite completely .
You can run original test, rerun failed and merge the results of both runs. If some tests are failed in the first run and then pass in the second, you will see that in the results.
There is often a need to re-execute a subset of tests, for example,
after fixing a bug in the system under test or in the tests
themselves. This can be accomplished by selecting test cases by names
(--test and --suite options), tags (--include and --exclude), or by
previous status (--rerunfailed or --rerunfailedsuites).
robot --output original.xml tests # first execute all tests
robot --rerunfailedsuites original.xml --output rerun.xml tests # then re-execute failing
rebot --merge original.xml rerun.xml # finally merge results
You can read more about this in here https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#merging-re-executed-tests
To your specific example, I am not sure that you can do that. But you can save the exit code of the run and evaluate it base on that
robot "your robot options" $#
if [ $? -eq 0 ]; then
"evaluation options when passed"
fi
else
"evaluation options when failed"
fi
When go test is ran it runs your files ending in _test.go by running the functions that start in the format TestXxx and use the (*t testing.T) module. I was wondering if each function in the _test.go file were ran concurrently or if it definitively ran each function separately? Does it create a go routine for each one? If it does create a go routine for each one, can I monitor the go routines in some way? Is it ever possible to do something like golibrary.GoRoutines() and get an instance for each one and monitor them some how or something like that?
Note: this question assumes you using the testing framework that comes with go (testing).
Yes, tests are executed as goroutines and, thus, executed concurrently.
However, tests do not run in parallel by default as pointed out by #jacobsa. To enable parallel execution you would have to call t.Parallel() in your test case and set GOMAXPROCS appropriately or supply -parallel N (which is set to GOMAXPROCS by default).
The simplest solution for your case when running tests in parallel would be to have a global slice for port numbers and a global atomically incremented index that serves as association between test and port from the slice. That way you control the port numbers and have one port for each test. Example:
import "sync/atomic"
var ports [...]uint64 = {10, 5, 55}
var portIndex uint32
func nextPort() uint32 {
return atomic.AddUint32(&portIndex, 1)
}
Yes, as other answers already pointed, tests inside one _test.go file are running in parallel only if you add t.Parallel() and run with -parallel tag.
BUT - by default the command go test runs in parallel tests for different packages, and default is the number of CPUs available (see go help build)
So to disable parallel execution you should run tests like this
go test -p 1 ./...
Yes if you add this :
t.Parallel() to your functions.
like this:
func TestMyFunction(t *testing.T) {
t.Parallel()
//your test code here
}
There are two aspects of "parallel tests" in go.
Testing of packages (folders) in parallel. Controlled by -p flag.
go help build:
-p n
the number of programs, such as build commands or
test binaries, that can be run in parallel.
The default is GOMAXPROCS, normally the number of CPUs available.
Testing of tests within a package. Controlled by -parallel flag, and tests need this enabled with t.Parallel().
go help testflag:
-parallel n
Allow parallel execution of test functions that call t.Parallel.
The value of this flag is the maximum number of tests to run
simultaneously; by default, it is set to the value of GOMAXPROCS.
Note that -parallel only applies within a single test binary.
The 'go test' command may run tests for different packages
in parallel as well, according to the setting of the -p flag
(see 'go help build').
For #1, you can observe this directly by having tests in multiple packages, and forcing them to be long running with a sleep. Then ps aux and grep for go running and you'll see separate processes in parallel.
Or run with -x detail flag. That pretty clearly shows what’s going on. The tests are launched once per package (e.g. given have *_test.go files in several package. Note I’m running go test -x ./... -v -count=1 |& ruby -pe 'print Time.now.strftime("[%Y-%m-%d %H:%M:%S] ")' |& tee temp.out with various configurations of -p=N -parallel=M, you can see in parallel them running:
[2022-05-09 17:22:26] $WORK/b065/understandinit.test -test.paniconexit0 -test.timeout=10m0s -test.parallel=1 -test.v=true -test.count=1
[2022-05-09 17:22:26] $WORK/b001/main.test -test.paniconexit0 -test.timeout=10m0s -test.parallel=1 -test.v=true -test.count=1
[2022-05-09 17:22:26] $WORK/b062/mainsub.test -test.paniconexit0 -test.timeout=10m0s -test.parallel=1 -test.v=true -test.count=1
You can run them concurrently by flagging the test with t.Parallel
and then run the test using the -parallel flag.
You can see other testing flags here
The go test command has support for the -c flag, described as follows:
-c Compile the test binary to pkg.test but do not run it.
(Where pkg is the last element of the package's import path.)
As far as I understand, generating a binary like this is the way to run it interactively using GDB. However, since the test binary is created by combining the source and test files temporarily in some /tmp/ directory, this is what happens when I run list in gdb:
Loading Go Runtime support.
(gdb) list
42 github.com/<username>/<project>/_test/_testmain.go: No such file or directory.
This means I cannot happily inspect the Go source code in GDB like I'm used to. I know it is possible to force the temporary directory to stay by passing the -work flag to the go test command, but then it is still a huge hassle since the binary is not created in that directory and such. I was wondering if anyone found a clean solution to this problem.
Go 1.5 has been released, and there is still no officially sanctioned Go debugger. I haven't had much success using GDB for effectively debugging Go programs or test binaries. However, I have had success using Delve, a non-official debugger that is still undergoing development: https://github.com/derekparker/delve
To run your test code in the debugger, simply install delve:
go get -u github.com/derekparker/delve/cmd/dlv
... and then start the tests in the debugger from within your workspace:
dlv test
From the debugger prompt, you can single-step, set breakpoints, etc.
Give it a whirl!
Unfortunately, this appears to be a known issue that's not going to be fixed. See this discussion:
https://groups.google.com/forum/#!topic/golang-nuts/nIA09gp3eNU
I've seen two solutions to this problem.
1) create a .gdbinit file with a set substitute-path command to
redirect gdb to the actual location of the source. This file could be
generated by the go tool but you'd risk overwriting someone's custom
.gdbinit file and would tie the go tool to gdb which seems like a bad
idea.
2) Replace the source file paths in the executable (which are pointing
to /tmp/...) with the location they reside on disk. This is
straightforward if the real path is shorter then the /tmp/... path.
This would likely require additional support from the compiler /
linker to make this solution more generic.
It spawned this issue on the Go Google Code issue tracker, to which the decision ended up being:
https://code.google.com/p/go/issues/detail?id=2881
This is annoying, but it is the least of many annoying possibilities.
As a rule, the go tool should not be scribbling in the source
directories, which might not even be writable, and it shouldn't be
leaving files elsewhere after it exits. There is next to nothing
interesting in _testmain.go. People testing with gdb can break on
testing.Main instead.
Russ
Status: Unfortunate
So, in short, it sucks, and while you can work around it and GDB a test executable, the development team is unlikely to make it as easy as it could be for you.
I'm still new to the golang game but for what it's worth basic debugging seems to work.
The list command you're trying to work can be used so long as you're already at a breakpoint somewhere in your code. For example:
(gdb) b aws.go:54
Breakpoint 1 at 0x61841: file /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.go, line 54.
(gdb) r
Starting program: /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.test
[snip: some arnings about BinaryCache]
Breakpoint 1, github.com/stellar/deliverator/aws.imageIsNewer (latest=0xc2081fe2d0, ami=0xc2081fe3c0, ~r2=false)
at /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.go:54
54 layout := "2006-01-02T15:04:05.000Z"
(gdb) list
49 func imageIsNewer(latest *ec2.Image, ami *ec2.Image) bool {
50 if latest == nil {
51 return true
52 }
53
54 layout := "2006-01-02T15:04:05.000Z"
55
56 amiCreationTime, amiErr := time.Parse(layout, *ami.CreationDate)
57 if amiErr != nil {
58 panic(amiErr)
This is just after running the following in the aws subdir of my project:
go test -c
gdb aws.test
As an additional caveat, it does seem very selective about where breakpoints can be placed. Seems like it has to be an expression but that conclusion is only via experimentation.
If you're willing to use tools besides GDB, check out godebug. To use it, first install with:
go get github.com/mailgun/godebug
Next, insert a breakpoint somewhere by adding the following statement to your code:
_ = "breakpoint"
Now run your tests with the godebug test command.
godebug test
It supports many of the parameters from the go test command.
-test.bench string
regular expression per path component to select benchmarks to run
-test.benchmem
print memory allocations for benchmarks
-test.benchtime duration
approximate run time for each benchmark (default 1s)
-test.blockprofile string
write a goroutine blocking profile to the named file after execution
-test.blockprofilerate int
if >= 0, calls runtime.SetBlockProfileRate() (default 1)
-test.count n
run tests and benchmarks n times (default 1)
-test.coverprofile string
write a coverage profile to the named file after execution
-test.cpu string
comma-separated list of number of CPUs to use for each test
-test.cpuprofile string
write a cpu profile to the named file during execution
-test.memprofile string
write a memory profile to the named file after execution
-test.memprofilerate int
if >=0, sets runtime.MemProfileRate
-test.outputdir string
directory in which to write profiles
-test.parallel int
maximum test parallelism (default 4)
-test.run string
regular expression to select tests and examples to run
-test.short
run smaller test suite to save time
-test.timeout duration
if positive, sets an aggregate time limit for all tests
-test.trace string
write an execution trace to the named file after execution
-test.v
verbose: print additional output
How can we list all the functions being called in an application. I tried using GDB but its backtrace list only upto the main function call.
I need deeper list i.e list of all the functions being called by the main function and the function being called from these called functions and so on.
Is there a way to get this in gdb? Or could you give me suggestions on how to get this?
How can we list all the functions being called in an application
For any realistically sized application, this list will have thousands of entries, which will probably make it useless.
You can find out all functions defined (but not necessarily called) in an application with the nm command, e.g.
nm /path/to/a.out | egrep ' [TW] '
You can also use GDB to set a breakpoint on each function:
(gdb) set logging on # collect trace in gdb.txt
(gdb) set confirm off # you wouldn't want to confirm every one of them
(gdb) rbreak . # set a breakpoint on each function
Once you continue, you'll hit a breakpoint for each function called. Use the disable and continue commands to move forward. I don't believe there is an easy way to automate that, unless you want to use Python scripting.
Already mentioned gprof is another good option.
You want a call graph. The tool that you want to use is not gdb, it's gprof. You compile your program with -pg and then run it. When it runs a file gmon.out will be produced. You then process this file with gprof and enjoy the output.
record function-call-history
https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html
This should be a great hardware accelerated possibility if you are one of the few people (2015) with a CPU that supports Intel Processor Tracing (Intel PT, intel_pt in /proc/cpuinfo).
GDB docs claim that it can produce output like:
(gdb) list 1, 10
1 void foo (void)
2 {
3 }
4
5 void bar (void)
6 {
7 ...
8 foo ();
9 ...
10 }
(gdb) record function-call-history /ilc
1 bar inst 1,4 at foo.c:6,8
2 foo inst 5,10 at foo.c:2,3
3 bar inst 11,13 at foo.c:9,10
Before using it you need to run:
start
record btrace
which is where a non capable CPU fails with:
Target does not support branch tracing.
CPU support is further discussed at: How to run record instruction-history and function-call-history in GDB?
Related threads:
how to trace function call in C?
Is there a compiler feature to inject custom function entry and exit code?
For embedded, you also consider JTAG and supporting hardware like ARM's DSTREAM, but x86 support does not seem very good: debugging x86 kernel using a hardware debugger
This question might need clarification to decide between what are currently 2 answers. Depends on what you need:
1) You need to know how many times each function is being called in straight list/graph format of functions matched with # of calls. This could lead to ambiguous/inconclusive results if your code is not procedural (i.e. functions calling other functions in a branch out structure without ambiguity of what is calling what). This is basic gprof functionality which requires recompilation with -pg flag.
2) You need a list of functions in the order in which they were called, this depends on your program which is the best/feasible option:
a) IF your program runs and terminates without runtime errors you can use gprof for this purpose.
b) ELSE option above using dbg with logging and break points is the left over option that I learned upon reading this.
3) You need to know not only the order but, for example, the function arguments for each call as well. My current work is simulations in physics of particle transport, so this would ABSOLUTELY be useful in tracking down where anomalous results are coming from... i.e. when the arguments getting passed around stop making sense. I imagine one way to do this is would be a variation on what Employed Russian did except using the following:
(gdb) info args
Logging the results of this command with every break point (set at every function call) gives the args of the current function.
With gdb, if you can find the most child function, you can list its all ancestors like this:
gdb <your-binary>
(gdb) b theMostChildFunction ## put breakpoint on the desired function
(gdb) r ## run the program
(gdb) bt ## backtrace starting from the breakpoint
Otherwise, on linux, you can use perf tool to trace programs and their function calls. The advantage of this, it is tracing all processes including child processes and also it shows usage percentages of the functions in the program.
You can install perf like this:
sudo apt install linux-tools-generic
sudo apt install linux-cloud-tools-generic
Before using perf you may also need to remove some kernel restrictions temporarily:
sudo sh -c 'echo 0 >/proc/sys/kernel/kptr_restrict'
sudo sh -c 'echo 0 >/proc/sys/kernel/perf_event_paranoid'
sudo sh -c 'echo 0 >/proc/sys/kernel/yama/ptrace_scope'
After this, you can run your program binary with perf like this:
perf record -g -s -a <your-binary-and-its-flags>
Then either you can look the output on terminal like this:
perf report
or on text file like this:
perf report -i perf.data > output.txt
vim output.txt
when you are recording the function calls with perf also you may want to filter kernel calls with --all-user flag:
perf record -g -s -a --all-user <your-binary-and-its-flags>
For further information you can look here: https://perf.wiki.kernel.org/index.php/Tutorial