We are presenting unit tests for a C++ project. Our goal is to cover up to 60% the whole project. The project uses a lot of code generated by Apache Thrift for communication between Client and Server.
Should we make unit tests for the generated code? If we don't introduce unit tests for the code, some coverage tools will be complaining that we are neglecting a big part of the project.
Does Apache Thrift provide those unit tests already?
There are a bunch of tests that run on the CI servers to ensure
the Thrift compiler can be built and works
the language bindings can be built and work
the language bindings operate as expected (including cross-lang tests)
In addition to the Apache Thrift Test Suite each language binding may or may not define additional tests to test specific things as needed, e.g. as shown here.
Regarding "is it necessary": First, we all make errors (except Linus of course). I would probably not go as far and test the entire thing again, but I would test those pieces in my own code that implement a certain behaviour using that 3rd party library.
I don't know Apache Thrift specifically, but it's probably safe to assume that any "big name" library/framework will have their own tests to try to ensure that it isn't fundamentally broken. However:
Those tests can't test your use of the library/framework. From a 30-second search, Thrift uses an Interface Definition Language, so if there are "problems" in the IDL code you write, Thrift may generate incorrect code (it may complain, but think the equivalent of undefined behaviour in C/C++).
Their tests won't test how your code uses the automatically generated code. You might be using it in ways it wasn't intended to be used, and so their "correct" code can still give the wrong results (for your application).
New releases of Thrift may behave differently than earlier versions, in ways that break your use of it. Assuming they're mentioned, scouring Release Notes for such changes is error-prone, so you may want some tests on features you rely on in the rest of your application. Especially important if you keep more-or-less automatically up-to-date with the latest version.
So, you probably don't need to unit-test it to the degree you would if you had written the code, but you will want tests near the generated code to detect problems with your use of it and/or changes in behaviour as new versions are released.
Related
Apparently, there are no coverage test modules in the ecosystem, and the only reference to something similar is the coverage tests in CommaIDE, which unfortunately are not present in the community (free) edition.
There seems to be some coverage at a lower level, MoarVM, but I don't see any way to do this easily in Raku modules. Is there maybe some simple language support for this?
The Comma IDE makes use of the MoarVM coverage output, which it parses, aggregates, and presents (using its model of the source code to figure out statement extents and which statements are coverable in order to generate the statistics).
The only other thing I'm aware of that currently exists to parse this output is this script. The MoarVM coverage support was originally developed in order to understand specification test coverage of the core built-ins, and the script makes a report of those. However, the mechanism that was put into MoarVM is actually more general, and so can be used to get the raw coverage data for any program. To my knowledge, however, the script I linked and Comma are the only tools built so far that analyze it.
I'm working on a C++ project in Visual Studio and wanted to test it using Google Test.
In order to do so, I built a solution which includes three projects:
Code project as a console application.
Google Test as a static library.
Tests project as a console application.
I wrote the code in the code project and a test in the test project.
When I tried building the solution, I got a lnk2019 error which stopped happening when I moved the classes to the tests project (but that's not where I want them to be).
I read that the best way to do so is by writing the classes as a library so I'll be able to use them in both (code and tests) projects.
Should I do that as a static library or as a DLL (and why)?
Other solutions are welcomed as well.
The key thing to understand here: you are getting things backwards.
You do not make this decision in order to get to "easier" testing. You first and foremost have to understand the "production requirements" towards your delivery. You create a static library, if that is the best for shipping your product. If not, you create a dynamic library.
For testing, you either deal with the chosen "delivery format"; or you think about a simple way that allows you to build a different thing for testing (although that is discouraged, as you want to test your product, nothing else).
Don't get me wrong: it is important to design for testing; but the decision you are asking for should be driven by your production requirements!
In other words: the "form factor" of your product should depend on product requirements, it should not be determined by your test setup.
We're just beginning a new project and we're keen to include testing from the ground up.
While we were looking at which unit test solution to use I came across Code Contracts which seem like they offer an easier way to check things like null parameter passing without having to write independent unit test methods.
One thing I am little confused about and makes me wary of investing heavily in Code Contract checks, is the fact that the analysis tool needs to be downloaded from DevLabs and isn't included in VS2012 by default.
What is the reason for this?
Additionally: It seems people are reporting that VS2012 support for Code Contract analysis seems flakey, why would be use Code Contracts if the analysis capabilities aren't very good?
I can't speak for VS2012, but it works perfectly fine for me in VS2010. There is a very minor conflict with Code Analysis in that a false alert is raised, but you simply switch off that Code Analysis rule and rely on the Code Contracts static checking (which is a lot more comprehensive).
Probably the best place to get the answer for your IDE integration is via the email address located on the Code Contracts Website, however I suspect that because its a research project, not having it as an "official" part of the IDE provides the ability for more regular updates.
We have a project with a lot of code, part of it is legacy.
As part of the work flow, every once in a while, all the functionality of the product is checked.
I wonder if there is a way to use this fact to dynamically check which parts of the code were never used? (The difficult part is the C++ code, the .Net and Java are more under control and have less legacy).
Also - are there dynamic dead code elimination tools are there that can work with lots of code and complex projects (i.e. ~1M lines)?
All the similar questions I found talked about static analysis which we all ready do.
Thank you!
You might want to look at the code coverage tools that are used in testing. The idea of these tools is that they instrument the code and after running the set of tests you know what lines of code were executed at least once and what lines were never executed. After that you can improve tests.
The same thing can be used to identify dead code in case if you have diverse enough execution environment.
I don't know what platform you are on but we have used Gcov with success if you're compiling with the gnu toolchain:
http://gcc.gnu.org/onlinedocs/gcc/Gcov.html
Background
I have an application written in native C++ over the course of several years that is around 60 KLOC. There are many many functions and classes that are dead (probably 10-15% like the similar Unix based question below asked). We recently began doing unit testing on all new code and applying it to modified code whenever possible. However, I would make a SWAG that we have less than 5% test coverage at the present moment.
Assumptions/Constraints
The method and/or tools must support:
Native (i.e. unmanaged) C++
Windows XP
Visual Studio 2005
Must not require user supplied test cases for coverage. (e.g. can't depend on unit tests to generate code coverage)
If the methods support more than these requirements, then great.
NOTE: We currently use the Professional edition of Visual Studio 2005, not the Team System. Therefore, using Team System might be a valid suggestion (I don't know, I've never used it) however I'm hoping it is not the only solution.
Why using unit tests for code coverage is problematic
I believe that it is impossible for a generic tool to find all the dead (e.g. unreachable code) in any arbitrary application with zero false positives (I think this would be equivalent to the Halting problem). However, I also believe it is possible for a generic tool to find many types of dead code that are highly probable to in fact be dead, like classes or functions which are never reference in the code by anything else.
By using unit tests to provide this coverage, you no longer using a generic algorithm and are thus increasing both the percentage of dead code you can detect and the probability that any hits are not false positives. Conversely, using unit tests could result in false negatives since the unit tests themselves might be the only thing exercising a given piece of code. Ideally, I would have regression testing that exercises all externally available methods, APIs, user controls, etc. which would serve as a baseline measurement of code coverage analysis to rule out certain methods from being false positives. Sadly however, I do not have this automated testing at the present time.
Since I have such a large code base with such a low test case coverage percentage however, I'm looking for something that could help without requiring huge amounts of time invested in writing test cases.
Question
How do you go about detecting dead code in an automated or semi-automated fashion in a native C++ application on the Windows platform with the Visual Studio 2005 development environment?
See Also
Dead code detection in legacy C/C++ project
I want tell the VC++ Compiler to compile all code. Can it be done?
Ask the linker to remove unreferenced objects (/OPT:REF). If you use function-level linking, and verbose linker output, the linker output will list every function it can prove is unused. This list may be far from complete, but you already have the tools needed.
We use Bullseye, and I can recommend it. It doesn't need to be run from a unit test environment, although that's what we do.
Use a code coverage tool against your unit test suite.