TeamCity Code Coverage of C++ code - unit-testing

We have a project using C#, C++/Cli and some native C++ code. We use TeamCity for building and testing.
We run the tests using vstest.console (VS2012 test runner).
For managed code, dotCover (which is integrated into TeamCity) is used for code coverage. However, it doesn't work with native C++ code (which is to be expected).
How do I get code coverage results our unit tests for the native C++ parts into TeamCity? Ideally, the solution would be free.


We use Bullseye Coverage for C++ code coverage. We then use the provided covxml tool to convert the binary coverage files into an XML file, which we then read out a bunch of useful attributes for function and conditional coverage (e.g. fn_total, fn_cov, cd_total, cd_cov) and provide these to TeamCity via the statistics service messages using the predefined coverage keys.
It was a bit of work to set up, but I think TeamCity still has no support for any C++ coverage tool, so our solution still works well for us years later.
Edit: I've uploaded the XML parsing code for our in-house tool to a Gist.

Related

mechanism to take code coverage using cppunit

I need to take the code coverage of my product code.
We use cppunit for unit testing. The complete code is in cpp.
Compilation of code takes place using make files.
Is there any way to take the code coverage using cppunit?
We do not use linux.
We use Keil for debugging and eclipse for code browsing.

CppUnit is not a code coverage tool. But it isn't incompatible with code coverage tools. So the answer is, you'd measure code coverage as with any other application. With Keil, you either need to be running in the simulator, or capture an instruction trace from your device using device-specific tools. Then you'd just load the trace into Keil's coverage analyzer.

unit test framework + code coverage tool for C++

We should select tool for our C++ code.
Is there a same tool or the tools of the same company for unit test framework
and code coverage tests?
Our aim if the both tool can work compatibly.
Sorry for my english
Thank you very much

For code coverage you can use the lcov tool and as Unit test Framework you can take a look at the Boost library

I used both CxxTest and GoogleTest and both do the job, with a main difference that the former is using pre-build script to generate the test driver code and the later uses static registration of tests which the driver later uses.
For coverage under Windows Bullseye Cover is a great product.

parser for developing c++ code coverage tool

I am developing a c++ code coverage tool. Are there any tools available for developing code coverage applications like parsers, etc.

I'd suggest that you look at the existing compilers and use their parsing abilities. For example there's clang/LLVM which already implements static code analysis of some form. Or you could use the built-in parser of the GNU Compilers/g++. IIRC, the new Visual Studio 2010 SDK also allows you to tap directly into the online compiler (that's also used for the new intellisense).

See my paper Branch Coverage Branch Coverage for Arbitrary Languages Made Easy which describes how to build test coverage tools using a program transformation system, by parsing source text, building an AST, and applying rewrite rules to insert test coverage probes, and prettyprinting the modified source code for compilation/execution/test probe data collection.
While the technique works with any system that can parse and prettyprint C++, as general rule these are hard to find, because parsing C++ is difficult. The paper particularly focuses on our our DMS Software Reengineering Toolkit, which has full C++ Front End, and for which we have built a C++ Test Coverage tool.

Code Coverage report on unit tests for Windows CE code using TFS 2010

I have a unique combination of platforms in my solution that has stumped me.
We need to generate code coverage statistics for unit tests that we're writing for our Windows CE code. We're using Visual Studio 2008 to write the CE code, of course, because VS 2010 doesn't support smart devices.
Unfortunately, MSTest won't instrument CE assemblies for collecting coverage data, so we're compiling our test assembly as a regular Win32 library and using it to call the CE code we need to test. This is working fine on a desktop installation of Visual Studio, but we also need to incorporate this into our nightly build on the server.
I have tried everything I can think of to get this to work. It's frustrating because I can manually do everything on my workstation, I just can't get TFS 2010 to do it for me automatically.
Is anybody else writing unit tests for Windows CE code? I can't believe I'm the only one!
Thanks in advance for whatever tips or tricks you might be able to offer...

If you are writing C++ or C# code, our SD Test Coverage Tools can be used to instrument your source code in a way that is compatible with embedded devices. These tools provide automated instrumentation of your source code, and instrumentation data collection procedures in source code so that you can control how the instrumentation data is collected and exactly how it is dumped, which is ideal for embedded software. A GUI display provides a programmer view of the coverage data imposed on the source code, and an XML report (which you can format at HTML using XSLT) provides summary data.

How can I measure CppUnit test coverage (on win32 and Unix)?

I have a very large code base that contains extensive unit tests (using CppUnit). I need to work out what percentage of the code is exercised by these tests, and (ideally) generate some sort of report that tells me on a per-library or per-file basis, how much of the code was exercised.
Here's the kicker: this has to run completely unnatended (eventually inside a continuous integration build), and has to be cross platform (well, WIN32 and *nix at least).
Can anyone suggest a tool, or set of tools that can help me do this? I can't change away from CppUnit (nor would I want to - it kicks ass), but otherwise I'm eager to hear any recommendations you might have.
Cheers,

Which tool should I use?
This article describes another developers frustrations searching for C++ code coverage tools. The author's final solution was Bullseye Coverage.
Bullseye Coverage features:
Cross Platform Support (win32, unix, and embedded), (supports linux gcc compilers and MSVC6)
Easy to use (up and running in a few hours).
Provides "best" metrics: Function Coverage and Condition/Decision Coverage.
Uses source code instrumentation.
As for hooking into your continuous integration, it depends on which CI solution you use, but you can likely hook the instrumentation / coverage measurement steps into the make file you use for automated testing.
Testing Linux vs Windows?
So long as all your tests run correctly in both environments, you should be fine measuring coverage on one or the other. (Though Bullseye appears to support both platforms). But why aren't you doing continuous integration builds in both environments?? If you deliver to clients in both environments then you need to be testing in both.
For that reason, it sounds like you might need to have two continuous build servers set up, one for a linux build and one for a windows build. Perhaps this can be easily accomplished with some virtualization software like vmware or virtualbox. You may not need to run code coverage metrics on both OSs, but you should definitely be running your unit tests on both.

If you can use GNU GCC as your complier, then the gcov tool works well. It's very easy to fully automate the whole process.

If you are using the GCC toolchain, gcov is going to get you source, functional, and branch coverage statistics. gcov works fine for MinGW and Cygwin. This will allow you to get coverage statistics as well as emitting instrumented source code that allows you to visualize unexecuted code.
However, if you really want to hit it out of the park with pretty reports, using gcov in conjunction with lcov is the way to go. lcov will give you bar reports scoped to files and directories, functional coverage statistics, and color coded source file browsing to show coverage (green means executed, red means not...).
lcov is easy on Linux, but may require some perl hacking on Cygwin. I personally have had some problems executing the scripts (lcov is implemented in perl) on Windows. I've gotten a hacked up version to work, but be forewarned.
Another approach is doing the gcov emit on windows, and doing the lcov post processing on Linux, where it will surely work out of the box.

Check out our SD C++ Test Coverage tool. It can be obtained for GCC, and for MSVC6.
It has low overhead probe data collection, a nice display of coverage data overlayed on your code, and complete report generation with rollups on coverage across the method/class/file/directory levels.
EDIT: Aug 2015: Now supports GCC5 and various MS dialects through Visual Studio 2015. To use these tools under Linux, you need Wine, but there the tools provide Linux-native sh scripting and a Linux/Java based UI, so the tool feels like a native Linux tool there.

I guess I should have specified the compiler - we're using gcc for Linux, and MSVC 6 (yeah I know, it's old, but it works (mostly) for us) for WIn32.
For that reasons, gcov won't work for our Win32 builds, and Bullseye won't work for our Linux builds.
Then again maybe I only need coverage in one OS...