I am developing a c++ code coverage tool. Are there any tools available for developing code coverage applications like parsers, etc.
I'd suggest that you look at the existing compilers and use their parsing abilities. For example there's clang/LLVM which already implements static code analysis of some form. Or you could use the built-in parser of the GNU Compilers/g++. IIRC, the new Visual Studio 2010 SDK also allows you to tap directly into the online compiler (that's also used for the new intellisense).
See my paper Branch Coverage Branch Coverage for Arbitrary Languages Made Easy which describes how to build test coverage tools using a program transformation system, by parsing source text, building an AST, and applying rewrite rules to insert test coverage probes, and prettyprinting the modified source code for compilation/execution/test probe data collection.
While the technique works with any system that can parse and prettyprint C++, as general rule these are hard to find, because parsing C++ is difficult. The paper particularly focuses on our our DMS Software Reengineering Toolkit, which has full C++ Front End, and for which we have built a C++ Test Coverage tool.
Related
We have a project using C#, C++/Cli and some native C++ code. We use TeamCity for building and testing.
We run the tests using vstest.console (VS2012 test runner).
For managed code, dotCover (which is integrated into TeamCity) is used for code coverage. However, it doesn't work with native C++ code (which is to be expected).
How do I get code coverage results our unit tests for the native C++ parts into TeamCity? Ideally, the solution would be free.
We use Bullseye Coverage for C++ code coverage. We then use the provided covxml tool to convert the binary coverage files into an XML file, which we then read out a bunch of useful attributes for function and conditional coverage (e.g. fn_total, fn_cov, cd_total, cd_cov) and provide these to TeamCity via the statistics service messages using the predefined coverage keys.
It was a bit of work to set up, but I think TeamCity still has no support for any C++ coverage tool, so our solution still works well for us years later.
Edit: I've uploaded the XML parsing code for our in-house tool to a Gist.
The clang C++ compiler claims to be built for, among other things, better IDE integration by providing an API for the IDE to use for tasks such as parsing the code.
So, are there are any good C++ IDE's that use clang to provide features such as semantic highlighting, refactoring, and finding and showing semantic errors in real-time?
I've been using Eclipse CDT, but its C++ parser is full of imperfections that cause the IDE to report a lot of annoying false positive errors in the code. I would like to have an IDE that reports an error if and only if the compiler would report the same error, hence my interest in an IDE that's built on a compiler's internals.
I'm primarily interested in cross-platform IDE's, although I wouldn't mind knowing about single-platform ones for Windows or Linux (so not Xcode), as long as they are FOSS (another reason why not Xcode).
Qt Creator is basing their next-gen code parsing and associated functionality on Clang:
https://www.qt.io/blog/2011/10/19/qt-creator-and-clang
Looks very, very promising!
have you tried clang complete?
if you're punk rock, then vim is enough ide ;)
i often work with xcode so... can't really share firsthand experience, but i knew of its existence.
gedit isn't really an IDE, but there is a plugin for it that provides code assistance using clang
It seems that CodeLite v3.5 starts supports Clang natively. However I haven't found is it possible to setup LLVM as backend.
A relevant new development in this area in the Language Server Protocol (LSP) project, which aims to be a language-agnostic API that allows editors / IDEs to be decoupled from backends that provide code intelligence / analysis.
There is ongoing work to create a clang-based C++ backend called Clangd.
There is also ongoing work on several editors / IDEs to support the LSP as a client.
Once the backend implementation matures, all editors supporting the LSP will, in principle, be able to leverage clang's capabilities as exposed through Clangd.
KDevelop now has clang based c and c++ support, including semantic analysis and autocomplete. It is primarily for linux but (as of October 2016) has a beta release out for windows and mac as well.
For emacs there are irony-mode and rtags that provide features such as auto-complete, on fly error checking and jump to symbol. When combined with cmake-ide they are very powerful tools and one well versed in emacs can be highly productive in this environment.
Not FOSS, but JetBrains (of IDEA and ReSharper fame) are building out their AppCode product into a full C++ IDE supporting Win/Linux/Mac and using clang.
Yes, really.
jucipp
~900 stars on GitHub in 2019Q2: https://github.com/cppit/jucipp
Now moved to GitLab: https://gitlab.com/cppit/jucipp
Clearly advertises libclang backend as a main feature.
What code coverage tools have you used with Symbian C++ and Maemo? What are the pros and cons of the tool you are using?
On Symbian I've used BullseyeCoverage and Testwell CTC++. Cannot really describe the pros/cons of them in detail. Both got the job done, eventually. Both needed some effort with setup and integration with an automated test suite. Both contained bugs that e.g. crashed the downstream compiler with slightly broken instrumented source code.
On Maemo, since the toolchain is GCC based, I'd guess gcov would be a good starting point. Though I haven't been working on Maemo much yet and haven't done any coverage measurement there.
See SD C++ Test Coverage for a tool that has extremely low overhead and works very well in embedded environments.
I have used Bullseye Coverage on Symbian and it is very good. The only problem is that it only runs on emulator and not hardware. Therefore you would not be able to get coverage metrics from a device or devboard. If your app runs on both hardware and emulator, this won't be a big deal (apart from the standard differences between running on emulator vs hardware). Also, as there are plans to replace the emulator with a proper hardware simulator, perhaps bullseye wouldn't be the best choice.
gcov can be used (and is used) in maemo platform and tool called lcov can then be used to generate "pretty" reports.
However, in order use gcov in maemo sdk, you need to disable compiler cache during the build time when you are creating binaries for coverage execution..
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
There's a lot of questions on here regarding various niche build needs (.NET, continuous integration, etc) but, of course, my niche need is different.
Rather than asking a very specific question right now, I'd like a survey of available build tools (such as make, ant, etc) so I can ask a follow up question more intelligently if needed.
In your answer, please include:
ONE build tool
Link to the main page about that tool
If you know some pros and cons (ie, runs on windows with cygwin, or .NET specific, etc) then list some (not required - the name and link are required)
If your build tool is already in an answer, comment on it adding pros/cons/limitations/experiences. Feel free to add as many answers as you like as long as they don't already exist - but don't put several build tools into one answer.
SCons
Build scripts are Python scripts. Supposed to work on Linux, Windows, Mac OS X.
NAnt
http://nant.sourceforge.net/
Similar to Ant, a build tool for .Net
Rake
Ruby version of make
Pros:
Clean idiomatic ruby syntax
Rake files are written in ruby, so leverages ruby itself rather than proprietary file format or XML
Ruby on Rails tightly integrated with it
Cons:
Requires ruby, which is not always available
No built-in packaging support/versioning/dependency management (see rubygems)
premake
Build script generator. Uses Lua to describe the build. It can generate Makefile or project files for Visual Studio, Code::Blocks, CodeLite, SharpDevelop, MonoDevelop, etc. Built-in support for C, C++, C#, including things like resources and ASP.NET. Mono support.
Jam family.
Jam
FTJam
BJam
BJam is based on FTJam, which in turn is based on Jam. Small, fast, portable. Automatic dependency analysis for C, C++. It is not a generator -- it does not generate any Makefile or other build files for secondary build systems.
Apache Maven
Pros:
Automatic dependency management
Utilizes convention over configuration (enforces consistent project directory structures)
Projects that use maven can be built in minutes
Excellent support for standard java/Java EE project builds
Works well in a team environment (supports "team" configurations)
Built in release/versioning concepts
Integrates with popular Java IDEs
Growing support in java OSS community
Cons:
Steep learning curve (esp. compared to Ant)
Poor online documentation (the new book is excellent though)
Sometimes surprising behavior
Very java-centric
XML configuration driven
GNU Make
http://www.gnu.org/software/make/
CMake
Cross-platform build system generator. It can generate Makefile or project files for Visual Studio, Eclipse CDT, KDevelop, Code::Blocks, etc. Automatic dependency analysis for C, C++, Fortran, Java.
Apache Ant
Pro:
many task/plugins
runs on many platforms
very mature
is supported by many IDEs, Continuous-Integration-Tools etc.
Con:
requires Java
FinalBuilder
Pros
Visual and GUI-oriented unlike ant or make
Lots of built-in actions
Script builder for your own actions
Integrates easily with cc.net
Cons
Only runs on Windows
Not free, but you get good value for your money
They also have a tool called Automise that does more system-oriented tasks.
Team City
Pros
All available here
Cons
Can be memory intensive
MSBuild
http://msdn.microsoft.com/en-us/library/wea2sca5(VS.80).aspx
CruiseControl/CruiseControl .Net
http://cruisecontrol.sourceforge.net/
Pros
Can use various build and TDD tools depending on version chosen
Automate when builds happen and what kind - full release build or changed code build
Can automatically run tests (nUnit, jUnit, Fitnesse, etc.) on a build to ensure that all tests still pass when new code is checked into the project.
Integrates with source control
Error reporting and notification when builds fail
Cons
Error reporting isn't always in a nice pretty to read format
Set up of projects should be well thought out and all projects monitored should use certain patterns to make integration setup more efficient.
PSake (pronounced "Sake")
Pros -
Powershell
Cons -
Powershell :)
This project is fairly new, looks interesting, and would be very powerful, since it's ".NET at the command line". Unfortunately, I don't know very many people who take the time to learn Powershell.
cons
Pros:
Uses Perl, if you like that sort of thing
Haven't used it otherwise :-)
Cons:
Hard to Google for!
Not actively developed?
UppercuT -
Some good explanations here: UppercuT
Pros -
Super easy to get started - Automated Builds as easy as (1) solution name, (2) source control path, (3) company name for most projects!!!
Limited knowledge of NAnt necessary.
Cons -
Only available for .NET
NUBuild (pronounced "New-Build")
This is the latest and pretty advanced .Net build tool (a very intelligent one) called NUBuild that allows you to build VB.Net and C# projects. Its extremely lightweight, open source and at the same time easy to setup and provides almost no-touch maintenance.
Easy to use command line interface
Ability to target all .Net framework version i.e. 1.1, 2.0, 3.0 and 3.5
Supports XML based configuration
Supports both project and file references
Automatically generates the “complete ordered build list” for a given project
Ability to detect and display circular dependencies
Perform parallel build
Ability to handle proxy assemblies
Easily integrated with Cruise-Control.Net continuous integration system
Version management capability
Notification feature
http://nubuild.codeplex.com/
It's best the build tool we have seen so far!
Bakefile
Bakefile is cross-platform, cross-compiler native makefiles generator. It takes compiler-independent description (XML) of build tasks as input and generates native makefile (autoconf's Makefile.in, Visual C++ project, bcc makefile etc.).
Bakefile's task is to generate native makefiles, so that people can keep using their favorite tools. There are other cross-platform make solutions, but they either aren't native and require the user to use unfamiliar tools (Boost.Build) or they are too limited (qmake).
I have a very large code base that contains extensive unit tests (using CppUnit). I need to work out what percentage of the code is exercised by these tests, and (ideally) generate some sort of report that tells me on a per-library or per-file basis, how much of the code was exercised.
Here's the kicker: this has to run completely unnatended (eventually inside a continuous integration build), and has to be cross platform (well, WIN32 and *nix at least).
Can anyone suggest a tool, or set of tools that can help me do this? I can't change away from CppUnit (nor would I want to - it kicks ass), but otherwise I'm eager to hear any recommendations you might have.
Cheers,
Which tool should I use?
This article describes another developers frustrations searching for C++ code coverage tools. The author's final solution was Bullseye Coverage.
Bullseye Coverage features:
Cross Platform Support (win32, unix, and embedded), (supports linux gcc compilers and MSVC6)
Easy to use (up and running in a few hours).
Provides "best" metrics: Function Coverage and Condition/Decision Coverage.
Uses source code instrumentation.
As for hooking into your continuous integration, it depends on which CI solution you use, but you can likely hook the instrumentation / coverage measurement steps into the make file you use for automated testing.
Testing Linux vs Windows?
So long as all your tests run correctly in both environments, you should be fine measuring coverage on one or the other. (Though Bullseye appears to support both platforms). But why aren't you doing continuous integration builds in both environments?? If you deliver to clients in both environments then you need to be testing in both.
For that reason, it sounds like you might need to have two continuous build servers set up, one for a linux build and one for a windows build. Perhaps this can be easily accomplished with some virtualization software like vmware or virtualbox. You may not need to run code coverage metrics on both OSs, but you should definitely be running your unit tests on both.
If you can use GNU GCC as your complier, then the gcov tool works well. It's very easy to fully automate the whole process.
If you are using the GCC toolchain, gcov is going to get you source, functional, and branch coverage statistics. gcov works fine for MinGW and Cygwin. This will allow you to get coverage statistics as well as emitting instrumented source code that allows you to visualize unexecuted code.
However, if you really want to hit it out of the park with pretty reports, using gcov in conjunction with lcov is the way to go. lcov will give you bar reports scoped to files and directories, functional coverage statistics, and color coded source file browsing to show coverage (green means executed, red means not...).
lcov is easy on Linux, but may require some perl hacking on Cygwin. I personally have had some problems executing the scripts (lcov is implemented in perl) on Windows. I've gotten a hacked up version to work, but be forewarned.
Another approach is doing the gcov emit on windows, and doing the lcov post processing on Linux, where it will surely work out of the box.
Check out our SD C++ Test Coverage tool. It can be obtained for GCC, and for MSVC6.
It has low overhead probe data collection, a nice display of coverage data overlayed on your code, and complete report generation with rollups on coverage across the method/class/file/directory levels.
EDIT: Aug 2015: Now supports GCC5 and various MS dialects through Visual Studio 2015. To use these tools under Linux, you need Wine, but there the tools provide Linux-native sh scripting and a Linux/Java based UI, so the tool feels like a native Linux tool there.
I guess I should have specified the compiler - we're using gcc for Linux, and MSVC 6 (yeah I know, it's old, but it works (mostly) for us) for WIn32.
For that reasons, gcov won't work for our Win32 builds, and Bullseye won't work for our Linux builds.
Then again maybe I only need coverage in one OS...