Can I capture simulator output to console in my testbench? - unit-testing

I have a testbench to test my VHDL device (DUT), but part of the DUT debug output is an ASSERT/REPORT message to the console, which I would like to check for correctness but I can't change the DUT. The only way I can think of is to post-process the output log file.
Is there any way of capturing the console output in the testbench, so I can check the DUT output directly?

I do this as part of the testbench. However, rather than Assert, I use OSVVM alerts, log, and print. OSVVM is both at osvvm.org and github.
Rather than Assert, I use AffirmIf for self-checking/result checking. I use AlertIf for parameter checking.
Step 1 is get OSVVM. Once you have the code, compile it using the script. In either Mentor or Aldec, run the script by doing:
vlib osvvm
vmap osvvm osvvm
do $PATH_TO_OSVVM/osvvm.do $PATH_TO_OSVVM
Use VHDL-2008 and include all of OSVVM in your program by doing:
library osvvm;
context osvvm.OsvvmContext;
Then rather than:
assert Data /= expected report "..." severity error;
Do:
AffirmIf(Data = Expected, "...") ;
Both assert and AffirmIf/AlertIf print. However, the advantage to AffirmIf/AlertIf is that internally it keeps a count of the errors and you can get a pass fail at the end of your test by doing:
ReportAlerts;
The next advantage of OSVVM AffirmIf/AlertIf/Log/Print is that if you want the results in a file, you simply do:
TranscriptOpen("./results/Test1.txt");
If you want to both print to the screen and a file, also do:
SetTranscriptMirror(TRUE);
That ought get you started. I will leave the rest to the user guides. Start by looking at both the AlertLog package user guide and the transcript package user guide.

Related

Differential code coverage for a simple example in ifort

I am interested in using the differential code coverage functionality in ifort. The documentation appears to address this thoroughly but I have failed to apply it to my reduced example. Heres what I have:
program test
integer :: userinput
print *, 'enter 1 or 0'
read *, userinput
if (userinput.eq.1) then
print *, 'You have entered ONE'
else
print *, 'You have not entered ONE'
end if
end program test
A simple program that can take one of two paths. If the user enters 1 then it goes into the if ... then statement, if the user enters 0 then it goes into the else... statement.
The goal of differential code coverage (as stated by intel docs) is as follows:
compare the profiles from two runs of an application: a reference run
and a new run identifying the code that is covered by the new run but
not covered by the reference run
So if we take a reference run where the user enters 0 and a new run where the user enters 1, the differential code coverage should be able to identify that the new run covers the if statement whereas the reference run does not (reference run goes into the else statement). I followed the docs as closely as possible. The source file is called test.f90. Here are the compile lines i'm using:
ifort test.f90 /Qcov-gen
Which generates PGOPTI.SPI, PGOPTI, test.exe and test.obj. I then run the executable and enter 0, I get the correct message "You have not entered ONE". This causes a .dyn file to be created (due to the Qcov-gen option). I then do the following:
profmerge
Which generates additional files pgopti.dpi, pgopti.dpi.lock. At this point I think I have enough material to generate my reference data. This I attempt using the following:
codecov -prj Project_Name -dpi pgopti.dpi -ref pgopti.dpi
Which generates html files similar to the ones displayed when code coverage is run in Visual Studio for Intel Fortran. I also get 100% code coverage which seems incorrect. The docs then show this command:
codecov -prj Project_Name -spi pgopti.spi -dpi pgopti.dpi
Which does not appear to provide an opportunity for a new run.
Could someone please explain how to do a simple differential code coverage on this particular example? I'm eventually trying to extrapolate this to a larger project but I'm trying to take baby steps to get there.

Fastest way to make console output "verbose" or not

I am making a small system and I want to be able to toggle "verbose" text output in the whole system.
I have made a file called globals.h:
namespace REBr{
extern bool console_verbose = false;
}
If this is true I want all my classes to print a message to the console when they are constructing, destructing, copying or doing pretty much anything.
For example:
window(string title="",int width=1280,int height=720):
Width(width),Height(height),title(title)
{
if(console_verbose){
std::cout<<"Generating window #"<<this->instanceCounter;
std::cout<<"-";
}
this->window=SDL_CreateWindow(title.c_str(),0,0,width,height,SDL_WINDOW_OPENGL);
if(console_verbose)
std::cout<<"-";
if(this->window)
{
this->glcontext = SDL_GL_CreateContext(window);
if(console_verbose)
std::cout<<".";
if(this->glcontext==NULL)
{
std::cout<<"FATAL ERROR IN REBr::WINDOW::CONSTR_OPENGLCONTEXT: "<<SDL_GetError()<<std::endl;
}
}
else std::cout<<"FATAL ERROR IN REBr::WINDOW::CONSTR_WINDOW: "<<SDL_GetError()<<std::endl;
if(console_verbose)
std::cout<<">done!"<<endl;
}
Now as you can see I have a lot of ifs in that constructor. And I REALLY dont want that since that will slow down my application. I need this to be as fast as possible without removing the "loading bar" (this helps me determine at which function the program stopped functioning).
What is the best/fastest way to accomplish this?
Everying in my system is under the namespace REBr
Some variants to achieve that:
Use some logger library. It is the best option as it gives you maximum flexibility and some useful experience ;) And you haven't to devise something. For example, look at Google GLOG.
Define some macro, allowing you to turn on/off all these logs by changing only the macro. But it isn't so easy to write such marco correctly.
Mark your conditional flag as constexpr. That way you may switch the flag and, depending on its value, compiler will optimise ifs in compiled program. But ifs will still be in code, so it looks kinda bulky.
Anyway, all these options require program recompilation. W/o recompilation it is impossible to achieve the maximum speed.
I often use a Logger class that supports debug levels. A call might look like:
logger->Log(debugLevel, "%s %s %d %d", timestamp, msg, value1, value2);
The Logger class supports multiple debug levels so that I can fine tune the debug output. This can be set at any time through the command line or with a debugger. The Log statement uses a variable length argument list much like printf.
Google's logging module is widely used in the industry and supports logging levels that you can set from the command line. For example (taken from their documentation)
VLOG(1) << "I'm printed when you run the program with --v=1 or higher";
VLOG(2) << "I'm printed when you run the program with --v=2 or higher";
You can find the code here https://github.com/google/glog and the documentation in the doc/ folder.

gcc gprof/gcov/other - how to get sequence of function calls/exits + control flow statements

BACKGROUND
We have testers for our embedded GUI product and when a tester declares "test failed", sometimes it's hard for us developers to reproduce the exact problem because we don't have the exact trace of what happened.
We do currently have a logging framework but us devs have to manually input those logging statements in the code which is fine . . . except when a hard-to-reproduce bug occurs and we didn't have a log statement at the 'right' location and then when we re-build, re-run the test with the same steps, we get a different result.
ISSUE
We would like a solution wherein the compiler produces extra instrumentation code that allows us to see the exact sequence of events including, at the minimum:
function enter/exit (already provided by -finstrument-functions)
control-flow statement enter i.e. entering if/else, which case statement we jumped to
The log would look like this:
int main() entered
if-line 5 entered
else-line 10 entered
void EventLoop() entered
. . .
Some additional nice-to-haves are
Parameter values on function entry AND exit (for pass-by-reference types)
Function return value
QUESTION
Are there any gcc tools or even paid tools that can do this instrumentation automatically?
You can either use gdb for that, and you can automate that (I've got a tool for that in the works, you find it here or you can try to use gcov.
The gcov implementation is so, that it loads the latest gcov data when you start the program. But: you can manually dump and load the data. If you call __gcov_flush it will dump the current data and reset the current state. However: if you do that several times it will always merge the data, so you'd also need to rename the gcov data file then.

GDB python script for bounded instruction tracing

I'm trying to write a GDB script to do instruction tracing in a bounded maner (i.e start addr and stop addr). Perhaps I'm failing at google but I cant seem to find this in existence already.
Here is my stab at it:
python
def start_logging():
gdb.execute("set logging on")
gdb.execute("while $eip != 0xBA10012E9")
gdb.execute("x/1i $eip")
gdb.execute("stepi")
gdb.execute(" end")
gdb.execute("set logging off")
gdb.execute("set pagination off")
gdb.execute("break *0xBA19912CF")
gdb.execute("command 1 $(start_logging())")
gdb.execute("continue")
In my mind this should set up a breakpoint then set the command to run when it hits. When the breakpoint hits it should single step through the code until the end address is hit and then it will turn off logging.
When I run this with gdb the application will break at the correct point but no commands are run.
What am I doing wrong? Sorry if this is the wrong way to go about this please let me know. I'm new to gdb scripting
I see a few odd things in here.
First, it looks like you are trying to split multi-line gdb commands across multiple calls to gdb.execute. I don't believe this will work. Certainly it isn't intended to work.
Second, there's no reason to try to do a "while" loop via gdb.execute. It's better to just do it directly in Python.
Third, I think the "command" line seems pretty wrong as well. I don't really get what it is trying to do, I guess call start_logging when the breakpoint is hit? And then continue? Well, it won't work as written.
What I would suggest is something like:
gdb.execute('break ...')
gdb.execute('run')
while gdb.parse_and_eval('$eip') != 0x...:
gdb.execute('stepi')
If you really want logging, either do the 'set logging' business or just instruct gdb.execute to return a string and log it from Python.

How to properly debug a binary generated by `go test -c` using GDB?

The go test command has support for the -c flag, described as follows:
-c Compile the test binary to pkg.test but do not run it.
(Where pkg is the last element of the package's import path.)
As far as I understand, generating a binary like this is the way to run it interactively using GDB. However, since the test binary is created by combining the source and test files temporarily in some /tmp/ directory, this is what happens when I run list in gdb:
Loading Go Runtime support.
(gdb) list
42 github.com/<username>/<project>/_test/_testmain.go: No such file or directory.
This means I cannot happily inspect the Go source code in GDB like I'm used to. I know it is possible to force the temporary directory to stay by passing the -work flag to the go test command, but then it is still a huge hassle since the binary is not created in that directory and such. I was wondering if anyone found a clean solution to this problem.
Go 1.5 has been released, and there is still no officially sanctioned Go debugger. I haven't had much success using GDB for effectively debugging Go programs or test binaries. However, I have had success using Delve, a non-official debugger that is still undergoing development: https://github.com/derekparker/delve
To run your test code in the debugger, simply install delve:
go get -u github.com/derekparker/delve/cmd/dlv
... and then start the tests in the debugger from within your workspace:
dlv test
From the debugger prompt, you can single-step, set breakpoints, etc.
Give it a whirl!
Unfortunately, this appears to be a known issue that's not going to be fixed. See this discussion:
https://groups.google.com/forum/#!topic/golang-nuts/nIA09gp3eNU
I've seen two solutions to this problem.
1) create a .gdbinit file with a set substitute-path command to
redirect gdb to the actual location of the source. This file could be
generated by the go tool but you'd risk overwriting someone's custom
.gdbinit file and would tie the go tool to gdb which seems like a bad
idea.
2) Replace the source file paths in the executable (which are pointing
to /tmp/...) with the location they reside on disk. This is
straightforward if the real path is shorter then the /tmp/... path.
This would likely require additional support from the compiler /
linker to make this solution more generic.
It spawned this issue on the Go Google Code issue tracker, to which the decision ended up being:
https://code.google.com/p/go/issues/detail?id=2881
This is annoying, but it is the least of many annoying possibilities.
As a rule, the go tool should not be scribbling in the source
directories, which might not even be writable, and it shouldn't be
leaving files elsewhere after it exits. There is next to nothing
interesting in _testmain.go. People testing with gdb can break on
testing.Main instead.
Russ
Status: Unfortunate
So, in short, it sucks, and while you can work around it and GDB a test executable, the development team is unlikely to make it as easy as it could be for you.
I'm still new to the golang game but for what it's worth basic debugging seems to work.
The list command you're trying to work can be used so long as you're already at a breakpoint somewhere in your code. For example:
(gdb) b aws.go:54
Breakpoint 1 at 0x61841: file /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.go, line 54.
(gdb) r
Starting program: /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.test
[snip: some arnings about BinaryCache]
Breakpoint 1, github.com/stellar/deliverator/aws.imageIsNewer (latest=0xc2081fe2d0, ami=0xc2081fe3c0, ~r2=false)
at /Users/mat/gocode/src/github.com/stellar/deliverator/aws/aws.go:54
54 layout := "2006-01-02T15:04:05.000Z"
(gdb) list
49 func imageIsNewer(latest *ec2.Image, ami *ec2.Image) bool {
50 if latest == nil {
51 return true
52 }
53
54 layout := "2006-01-02T15:04:05.000Z"
55
56 amiCreationTime, amiErr := time.Parse(layout, *ami.CreationDate)
57 if amiErr != nil {
58 panic(amiErr)
This is just after running the following in the aws subdir of my project:
go test -c
gdb aws.test
As an additional caveat, it does seem very selective about where breakpoints can be placed. Seems like it has to be an expression but that conclusion is only via experimentation.
If you're willing to use tools besides GDB, check out godebug. To use it, first install with:
go get github.com/mailgun/godebug
Next, insert a breakpoint somewhere by adding the following statement to your code:
_ = "breakpoint"
Now run your tests with the godebug test command.
godebug test
It supports many of the parameters from the go test command.
-test.bench string
regular expression per path component to select benchmarks to run
-test.benchmem
print memory allocations for benchmarks
-test.benchtime duration
approximate run time for each benchmark (default 1s)
-test.blockprofile string
write a goroutine blocking profile to the named file after execution
-test.blockprofilerate int
if >= 0, calls runtime.SetBlockProfileRate() (default 1)
-test.count n
run tests and benchmarks n times (default 1)
-test.coverprofile string
write a coverage profile to the named file after execution
-test.cpu string
comma-separated list of number of CPUs to use for each test
-test.cpuprofile string
write a cpu profile to the named file during execution
-test.memprofile string
write a memory profile to the named file after execution
-test.memprofilerate int
if >=0, sets runtime.MemProfileRate
-test.outputdir string
directory in which to write profiles
-test.parallel int
maximum test parallelism (default 4)
-test.run string
regular expression to select tests and examples to run
-test.short
run smaller test suite to save time
-test.timeout duration
if positive, sets an aggregate time limit for all tests
-test.trace string
write an execution trace to the named file after execution
-test.v
verbose: print additional output