BACKGROUND
We have testers for our embedded GUI product and when a tester declares "test failed", sometimes it's hard for us developers to reproduce the exact problem because we don't have the exact trace of what happened.
We do currently have a logging framework but us devs have to manually input those logging statements in the code which is fine . . . except when a hard-to-reproduce bug occurs and we didn't have a log statement at the 'right' location and then when we re-build, re-run the test with the same steps, we get a different result.
ISSUE
We would like a solution wherein the compiler produces extra instrumentation code that allows us to see the exact sequence of events including, at the minimum:
function enter/exit (already provided by -finstrument-functions)
control-flow statement enter i.e. entering if/else, which case statement we jumped to
The log would look like this:
int main() entered
if-line 5 entered
else-line 10 entered
void EventLoop() entered
. . .
Some additional nice-to-haves are
Parameter values on function entry AND exit (for pass-by-reference types)
Function return value
QUESTION
Are there any gcc tools or even paid tools that can do this instrumentation automatically?
You can either use gdb for that, and you can automate that (I've got a tool for that in the works, you find it here or you can try to use gcov.
The gcov implementation is so, that it loads the latest gcov data when you start the program. But: you can manually dump and load the data. If you call __gcov_flush it will dump the current data and reset the current state. However: if you do that several times it will always merge the data, so you'd also need to rename the gcov data file then.
Related
I have a testbench to test my VHDL device (DUT), but part of the DUT debug output is an ASSERT/REPORT message to the console, which I would like to check for correctness but I can't change the DUT. The only way I can think of is to post-process the output log file.
Is there any way of capturing the console output in the testbench, so I can check the DUT output directly?
I do this as part of the testbench. However, rather than Assert, I use OSVVM alerts, log, and print. OSVVM is both at osvvm.org and github.
Rather than Assert, I use AffirmIf for self-checking/result checking. I use AlertIf for parameter checking.
Step 1 is get OSVVM. Once you have the code, compile it using the script. In either Mentor or Aldec, run the script by doing:
vlib osvvm
vmap osvvm osvvm
do $PATH_TO_OSVVM/osvvm.do $PATH_TO_OSVVM
Use VHDL-2008 and include all of OSVVM in your program by doing:
library osvvm;
context osvvm.OsvvmContext;
Then rather than:
assert Data /= expected report "..." severity error;
Do:
AffirmIf(Data = Expected, "...") ;
Both assert and AffirmIf/AlertIf print. However, the advantage to AffirmIf/AlertIf is that internally it keeps a count of the errors and you can get a pass fail at the end of your test by doing:
ReportAlerts;
The next advantage of OSVVM AffirmIf/AlertIf/Log/Print is that if you want the results in a file, you simply do:
TranscriptOpen("./results/Test1.txt");
If you want to both print to the screen and a file, also do:
SetTranscriptMirror(TRUE);
That ought get you started. I will leave the rest to the user guides. Start by looking at both the AlertLog package user guide and the transcript package user guide.
I am interested in using the differential code coverage functionality in ifort. The documentation appears to address this thoroughly but I have failed to apply it to my reduced example. Heres what I have:
program test
integer :: userinput
print *, 'enter 1 or 0'
read *, userinput
if (userinput.eq.1) then
print *, 'You have entered ONE'
else
print *, 'You have not entered ONE'
end if
end program test
A simple program that can take one of two paths. If the user enters 1 then it goes into the if ... then statement, if the user enters 0 then it goes into the else... statement.
The goal of differential code coverage (as stated by intel docs) is as follows:
compare the profiles from two runs of an application: a reference run
and a new run identifying the code that is covered by the new run but
not covered by the reference run
So if we take a reference run where the user enters 0 and a new run where the user enters 1, the differential code coverage should be able to identify that the new run covers the if statement whereas the reference run does not (reference run goes into the else statement). I followed the docs as closely as possible. The source file is called test.f90. Here are the compile lines i'm using:
ifort test.f90 /Qcov-gen
Which generates PGOPTI.SPI, PGOPTI, test.exe and test.obj. I then run the executable and enter 0, I get the correct message "You have not entered ONE". This causes a .dyn file to be created (due to the Qcov-gen option). I then do the following:
profmerge
Which generates additional files pgopti.dpi, pgopti.dpi.lock. At this point I think I have enough material to generate my reference data. This I attempt using the following:
codecov -prj Project_Name -dpi pgopti.dpi -ref pgopti.dpi
Which generates html files similar to the ones displayed when code coverage is run in Visual Studio for Intel Fortran. I also get 100% code coverage which seems incorrect. The docs then show this command:
codecov -prj Project_Name -spi pgopti.spi -dpi pgopti.dpi
Which does not appear to provide an opportunity for a new run.
Could someone please explain how to do a simple differential code coverage on this particular example? I'm eventually trying to extrapolate this to a larger project but I'm trying to take baby steps to get there.
I have some code that calculates the price of a stock option using Monte Carlo and returns a discounted price. The final few lines of the relevant method look like this:
if(payoffType == pt.LongCall or payoffType == pt.LongPut):
discountedPrice=discountedValue
elif(payoffType == pt.ShortCall or payoffType == pt.ShortPut):
discountedPrice=(-1.0)*discountedValue
else:
raise Exception
#endif
print "dv:", discountedValue, " px:", discountedPrice
return discountedPrice
At a higher level of the program, I create four pricers, which are passed to instances of a portfolio class that calls the price() method on the pricer it has received.
When I set the breakpoint on the if statement or the print statement, the breakpoints work as expected. When I set the breakpoint on the return statement, the breakpoint is interpreted correctly on the first pass through the pricing code, but then skipped on subsequent passes.
On occasion, if I have set a breakpoint somewhere in the flow of execution between the first pass through the pricing code and the second pass, the breakpoint will be picked up.
I obviously have a workaround, but I'm curious if anyone else has observed this behavior in the PyDev debugger, and if so, does anyone know the root cause?
The issues I know of are:
If you have a StackOverflowError anywhere in the code, Python will disable the tracing that the debugger uses.
I know there are some issues with asynchronous code which could make the debugger break.
A workaround is using a programmatic breakpoint (i.e.: pydevd.settrace -- the remote debugger session: http://www.pydev.org/manual_adv_remote_debugger.html has more details on it) -- it resets the tracing even if Python broke it in a stack overflow error and will always be hit to (the issue on asynchronous code is that the debugger tries to run with untraced threads, but sometimes it's not able to restore it on some conditions when dealing with asynchronous code).
The go test command has support for the -c flag, described as follows:
-c Compile the test binary to pkg.test but do not run it.
(Where pkg is the last element of the package's import path.)
As far as I understand, generating a binary like this is the way to run it interactively using GDB. However, since the test binary is created by combining the source and test files temporarily in some /tmp/ directory, this is what happens when I run list in gdb:
Loading Go Runtime support.
(gdb) list
42 github.com/<username>/<project>/_test/_testmain.go: No such file or directory.
This means I cannot happily inspect the Go source code in GDB like I'm used to. I know it is possible to force the temporary directory to stay by passing the -work flag to the go test command, but then it is still a huge hassle since the binary is not created in that directory and such. I was wondering if anyone found a clean solution to this problem.
Go 1.5 has been released, and there is still no officially sanctioned Go debugger. I haven't had much success using GDB for effectively debugging Go programs or test binaries. However, I have had success using Delve, a non-official debugger that is still undergoing development: https://github.com/derekparker/delve
To run your test code in the debugger, simply install delve:
go get -u github.com/derekparker/delve/cmd/dlv
... and then start the tests in the debugger from within your workspace:
dlv test
From the debugger prompt, you can single-step, set breakpoints, etc.
Give it a whirl!
Unfortunately, this appears to be a known issue that's not going to be fixed. See this discussion:
https://groups.google.com/forum/#!topic/golang-nuts/nIA09gp3eNU
I've seen two solutions to this problem.
1) create a .gdbinit file with a set substitute-path command to
redirect gdb to the actual location of the source. This file could be
generated by the go tool but you'd risk overwriting someone's custom
.gdbinit file and would tie the go tool to gdb which seems like a bad
idea.
2) Replace the source file paths in the executable (which are pointing
to /tmp/...) with the location they reside on disk. This is
straightforward if the real path is shorter then the /tmp/... path.
This would likely require additional support from the compiler /
linker to make this solution more generic.
It spawned this issue on the Go Google Code issue tracker, to which the decision ended up being:
https://code.google.com/p/go/issues/detail?id=2881
This is annoying, but it is the least of many annoying possibilities.
As a rule, the go tool should not be scribbling in the source
directories, which might not even be writable, and it shouldn't be
leaving files elsewhere after it exits. There is next to nothing
interesting in _testmain.go. People testing with gdb can break on
testing.Main instead.
Russ
Status: Unfortunate
So, in short, it sucks, and while you can work around it and GDB a test executable, the development team is unlikely to make it as easy as it could be for you.
I'm still new to the golang game but for what it's worth basic debugging seems to work.
The list command you're trying to work can be used so long as you're already at a breakpoint somewhere in your code. For example:
(gdb) b aws.go:54
Breakpoint 1 at 0x61841: file /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.go, line 54.
(gdb) r
Starting program: /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.test
[snip: some arnings about BinaryCache]
Breakpoint 1, github.com/stellar/deliverator/aws.imageIsNewer (latest=0xc2081fe2d0, ami=0xc2081fe3c0, ~r2=false)
at /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.go:54
54 layout := "2006-01-02T15:04:05.000Z"
(gdb) list
49 func imageIsNewer(latest *ec2.Image, ami *ec2.Image) bool {
50 if latest == nil {
51 return true
52 }
53
54 layout := "2006-01-02T15:04:05.000Z"
55
56 amiCreationTime, amiErr := time.Parse(layout, *ami.CreationDate)
57 if amiErr != nil {
58 panic(amiErr)
This is just after running the following in the aws subdir of my project:
go test -c
gdb aws.test
As an additional caveat, it does seem very selective about where breakpoints can be placed. Seems like it has to be an expression but that conclusion is only via experimentation.
If you're willing to use tools besides GDB, check out godebug. To use it, first install with:
go get github.com/mailgun/godebug
Next, insert a breakpoint somewhere by adding the following statement to your code:
_ = "breakpoint"
Now run your tests with the godebug test command.
godebug test
It supports many of the parameters from the go test command.
-test.bench string
regular expression per path component to select benchmarks to run
-test.benchmem
print memory allocations for benchmarks
-test.benchtime duration
approximate run time for each benchmark (default 1s)
-test.blockprofile string
write a goroutine blocking profile to the named file after execution
-test.blockprofilerate int
if >= 0, calls runtime.SetBlockProfileRate() (default 1)
-test.count n
run tests and benchmarks n times (default 1)
-test.coverprofile string
write a coverage profile to the named file after execution
-test.cpu string
comma-separated list of number of CPUs to use for each test
-test.cpuprofile string
write a cpu profile to the named file during execution
-test.memprofile string
write a memory profile to the named file after execution
-test.memprofilerate int
if >=0, sets runtime.MemProfileRate
-test.outputdir string
directory in which to write profiles
-test.parallel int
maximum test parallelism (default 4)
-test.run string
regular expression to select tests and examples to run
-test.short
run smaller test suite to save time
-test.timeout duration
if positive, sets an aggregate time limit for all tests
-test.trace string
write an execution trace to the named file after execution
-test.v
verbose: print additional output
At the moment I'm using an eclipse-like IDE and the corresponding debug perspective, that most of you are probably familiar with. While debugging code I quite often find myself stepping through many lines of code and observing variables and double checking if everything is as it is supposed to be.
But suppose there is something like this:
1. important line, e.g. generating a new object;
2. another important line, e.g. some tricky class method;
3. for (int i = 0; i < some_limit; ++i)
4. some_array[i]++;
5. more important stuff;
Obviously I'm interested in what happens in lines 1,2 and 5 (I know this is a poor example, but please bear with me for a little while longer) but I don't want to step through all hundreds (or even thousands) of iterations of lines 3/4.
So, finally, my question: Is there some way to step directly over the for-cycle? What I do right now is set a new breakpoint at line 5 and let the program run as soon as I hit line 3 and I believe this is not an optimal solution.
edit: The eclipse implementation of what ks1322 proposed is called "Run to line" and is mapped to ctrl-r
Use until command instead of next.
From gdb documentation:
Continue running until a source line past the current line, in the
current stack frame, is reached. This command is used to avoid single
stepping through a loop more than once.
If you will use until instead of next, gdb will step over loops only once, which almost exactly what you want.