Currently we use gcov with our testing suite for Linux C++ application and it does a good job at measuring line coverage.
Can gcov produce function/method coverage report in addition to line coverage?
Looking at the parameters gcov accepts I do not think it is possible, but I may be missing something. Or, probably, is there any other tool that can produce function/method coverage report out of statistics generated by gcc?
Update: By function/method coverage I mean percentage of functions that get executed during tests.
I guess what you mean is the -f option, which will give you the percentage of lines covered per function. There is an interesting article about gcov at Dr. Dobb's which might be helpful. If "man gcov" doesn't show the -f flag, check if you have a reasobably recent version of the gcc suite.
Edit: to get the percentage of functions not executed you can simply parse through the function coverage output, as 0.00% coverage should be pretty much equivalent to not called. This small script prints the percentage of functions not executed:
#!/bin/bash
if test -z "$1"
then
echo "First argument must be function coverage file"
else
notExecuted=`cat $1 | grep "^0.00%" | wc -l`
executed=`cat $1 | grep -v "^0.00%" | wc -l`
percentage=$(echo "scale=2; $notExecuted / ($notExecuted + $executed) * 100" |bc)
echo $percentage
fi
We have started to use gcov and lcov together. The results from lcov do include the percentage of functions that are executed for the "module" you're looking at.
EDIT: The module can go from directories down to files.
I also want to add that if you are already using the GNU compiler tools, then gcov/lcov won't be too difficult for you to get running and the results it produces are very impressive.
The lcov utility is nice, and we use it. But I'm not sure if you need it for what you want.
We
Use ctags (wikipedia; sourceforge) to find all the functions declared in the relevant header files.
Run GCOV to get line coverage for every function in the binary.
Compare the list of functions from 1 & 2 to produce "Functions Called" / "Functions Available".
We call it "API coverage" since we apply step #1 only to public API headers. But you can do it on all headers or only a subset as you choose. I think the ratio we produce in this manner is the ratio you are looking for.
Related
I manage a quite large open-source project with many unit-tests (~200 files) and passing all tests is quite time consuming for the continuous integration. We use cmake/ctest/Catch2 for the unit-test framework.
Is there a way to tell cmake/ctest to only build and run a random subset of the unit tests (e.g. just 30%) ?
When iterating with several commits on the code for a given feature, the probability that all tests where checked tends to one, but each individual commit would be way faster.
Obviously, this ratio would be set to 100% when preparing a PR or a release.
ALTETRNATIVE
Finally, I came up with a cmake solution just by creating a new add_test() function that activate a test upon a random test:
function(my_add_test test_file) #optional_avoid_add_test
string(RANDOM LENGTH 2 ALPHABET "0123456789" _random)
if (${_random} LESS ${THRESHOLD_RANDOM_TESTING})
add_executable(${test_file} ${test_file}.cpp)
add_test(${test_file} ${test_file})
endif()
endfunction()
In my main cmake I have the global variable (that can be set at cmake CLI/GUI)
SET(THRESHOLD_RANDOM_TESTING "100" CACHE INTERNAL "~% of unit tests to build and run.")
Each time I regenerate the project, a new random selection is constructed.
Is there a way to tell cmake/ctest
Yes. Get all test, shuffle, get 30% of them and pass them to back to ctest.
Looks like fun, in linux shell would be just:
ctest -N | sed -n 's/ Test #[0-9]\+: //p' | { tmp=$(cat); cnt=$(wc -l <<<"$tmp"); shuf -n "$((cnt * 30 / 100))" <<<"$tmp"; }
I meant an automatic way to do that.
No, there is not.
I test my code with go test ./... -v -short.
Unfortunately, -v only prints out each test as it happens, but does not leave a summary of the results at the bottom like in Java. This means that if any test failed somewhere at the top, I have to scroll up and look for the word FAIL or search for it in a text editor.
The -failfast flag isn't helping either because some of my tests still get printed after the first test failure for some reason.
I don't really care if tests get run after the initial test failure. I just want to be able to easily tell if any test failed, preferably in just one place (e.g. a summary of how many tests passed or failed, or by seeing a flag if all tests passed or not).
Is there a way to easily tell if there was a test failure because I don't want to accidentally continue coding if I still have test failures.
I'm on Windows 10 64-bit.
UPDATE: Many thanks to #icza for the findstr tip. I later realized that I also wanted to see the error descriptions along with the test failures, but did not want to run go test twice. This is what I came up with for CMD (does not work on Powershell):
go test ./... -v -short > test-results.txt & findstr "FAIL _test" test-results.txt
Now findstr should report test failures as well as error descriptions. And if you want to see the full test results, simply open test-results.txt.
Failing tests are indicated with FAIL in the output. So all you have to do is filter the output for that word.
On Unix systems:
go test ./... |grep FAIL
On Windows:
go test ./... |findstr FAIL
Note that this is purely text processing, it doesn't know anything about go tests and their results. This means you might get "false positives" if a test outputs the word FAIL even if it succeeds. But in practice, this pretty much does the job you want.
A more sophisticated and more accurate way to achieve this would be to pass -json flag to go test, so it generates JSON output, which you can process with a program (e.g. written in Go itself). Failing tests are indicated with a JSON object having an "Action":"fail" field, e.g.
{"Time":"2019-03-01T12:06:21.108544405+01:00","Action":"fail",
"Package":"some/package","Test":"TestSomething","Elapsed":0.01}
And even if you don't want to write a program for this, filtering the JSON output leaves less chance for false positive (filtering for "Action":"fail"):
Unix:
go test ./... -json |grep '"Action":"fail"'
Windows:
go test ./... -json |findstr /C:"\"Action\":\"fail\""
I found it painless to install gotestsum and get the neat summary at the end.
go install gotest.tools/gotestsum#latest
gotestsum --format testname # Or dots
An alternative, if you only care about the count is:
go test |grep FAIL |wc -l
I'm currently writing unit tests for a Qt project. I wanted to use the statistics provided in Jenkins through the Cobertura plugin (underneath gcov is used to get the stats).
:~$ gcov -v
gcov 5.4.0 20160609
:~$ gcc -v
gcc version 5.4.0
However after I looked at the table (see below) I was really surprised to see the poor coverage especially of conditionals. For the first one (see Coverage Breakdown by File) I thought I was actually done, since the code has only three ifs(each with a single condition) and my tests covers all (checked this also through debugging just to make sure). So I am really confused what these numbers actually mean and how to interpret them in order to make my unit tests better.
I've even started thinking that some of the poor results might be due to the use of Qt since it's not exactly pure C++ and all "extras" (slots, signals, MOC files etc.) might be something that gcov can't handle properly.
Checking the annotated source listing with displayed red/green coverage markings should help.
The numbers inside the coloured bars are line counts, hence 47/108 means that 47 lines of code out of the 108 lines that are controlled by conditionals, have coverage.
For each conditional you need at least two unit tests: one for each branch.
If there are && or || in the conditions - or anywhere else (eg a logical expression) then each combination must be exercised to achieve 100%.
Also don't forget
a = (j == 0) ? c : d;
requires (at least) two tests !
Also, if using C++, see Why gcc 4.1 + gcov reports 100% branch coverage and newer (4.4, 4.6, 4.8) reports 50% for "p = new class;" line?
I am interested in testing speed of some function calls from the code written in C/C++. I searched and I was directed to use Valgrind platform with Callgrind tool.
I have briefly read the manual, but I am still wondering how I can utilize the functionality of the tool to evaluate the time of my function runtime speed.
I was wondering if I could get some pointers how I can achieve my goal.
Any help would be appreciated.
Compile your program with debug symbols (e.g. GDB symbols works fine, which are activated with the "-ggdb" flag).
If you are executing your program like this:
./program
Then run it with Valgrind+Callgrind with this command:
valgrind --tool=callgrind ./program
Callgrind will then produce a file called callgrind.out.1234 (1234 is the process ID and will probably be different when you run). Open this file with:
cg_annotate callgrind.out.1234
You may want to use grep to extract your function name. In the left column the number of instructions used for that function is displayed. Functions that use a comparatively low amount of instructions will be ignored, though.
If you want to see the output some nice graphics, I would recommend you to install KCachegrind.
I am comparing two almost identical folders which include hidden .svn folders which should be ignored and I want to continually quickly compare the folders as some files are patched to compared the difference without checking the unchanged matching files again.
edit:
Because there are so many options I'm interested in a solution that clearly exploits the knowledge from the previous compare because any other solution is not really feasable when doing repeated comparisons.
If you are willing to spend a bit of money, Beyond Compare is a pretty powerful diffing tool that can do folder based diffing.
Beyond Compare
I personally use WinMerge and find it very useful. It has filters that exclude svn file. Under linux i prefer Meld.
One option would be to use rsync. Something like:
rsync -n -r -v -C dir_a dir_b
The -n option does a dry-run so no files will be modified. -r does a recursive comparison. Optionally turn on verbose mode with -v. (You could use -i to itemize the changes instead of -v.) To ignore commonly ignored files such as .svn/ use -C.
This should be faster than a simple diff as I read the rsync manpage:
Rsync finds files that need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed in size
or in last-modified time. Any changes in the other preserved
attributes (as requested by options) are made on the destination file
directly when the quick check indicates that the file's data does not
need to be updated.
Since the "quick check" algorithm does not look at file contents directly, it might be fooled. In that case, the -c option, which performs a checksum instead, may be needed. It is likely to be faster than an ordinary diff.
In addition, if you plan on syncing the directories at some point, this is a good tool for that job as well.
Not foolproof, but you could just compare the timestamps.
Use total commander ! All the cool developers use it :)
If you are on linux or some variant, you should be able to do:
prompt$ diff -r dir1 dir2 --exclude=.svn
The -r forces recursive lookups. There are a bunch of switches to ignore stuff like whitespace etc.