I'm new to C++ but have experience with Java. I'm writing an Arduino project which I'm testing using gtest. I'd like these tests to fail if I've made a mistake with memory management. I've found that AddressSanitizer and Valgrind are candidates for detecting leaks at runtime. It looks like the PlatformIo 'check' capability isn't as comprehensive. Since I run my unit tests repeatedly (currently a run takes ~5 seconds) I'm leaning towards AddressSanitizer for my memory validation.
I did some searches on Google and didn't find any tutorials. It could be this is so basic everyone knows how to do it, or that PIO doesn't need special configuration. Either way, I'm hoping someone has done this and can share a reference or explanation of the basic process. Thanks!
Update:
I got some guidance that it makes sense to test the general functionality before the target specific functionality. That makes tons of sense. For me this translates to having an entry in my Platform.ini like this:
[env]
test_framework = googletest
[env:native]
platform = native
build_src_filter = ...
build_src_filter excludes my Arduino specific code. I'm hoping to find a way to catch leaks in my native configuration. Ideally I can do this within the Platform.io experience.
Related
We're just beginning a new project and we're keen to include testing from the ground up.
While we were looking at which unit test solution to use I came across Code Contracts which seem like they offer an easier way to check things like null parameter passing without having to write independent unit test methods.
One thing I am little confused about and makes me wary of investing heavily in Code Contract checks, is the fact that the analysis tool needs to be downloaded from DevLabs and isn't included in VS2012 by default.
What is the reason for this?
Additionally: It seems people are reporting that VS2012 support for Code Contract analysis seems flakey, why would be use Code Contracts if the analysis capabilities aren't very good?
I can't speak for VS2012, but it works perfectly fine for me in VS2010. There is a very minor conflict with Code Analysis in that a false alert is raised, but you simply switch off that Code Analysis rule and rely on the Code Contracts static checking (which is a lot more comprehensive).
Probably the best place to get the answer for your IDE integration is via the email address located on the Code Contracts Website, however I suspect that because its a research project, not having it as an "official" part of the IDE provides the ability for more regular updates.
We have a project with a lot of code, part of it is legacy.
As part of the work flow, every once in a while, all the functionality of the product is checked.
I wonder if there is a way to use this fact to dynamically check which parts of the code were never used? (The difficult part is the C++ code, the .Net and Java are more under control and have less legacy).
Also - are there dynamic dead code elimination tools are there that can work with lots of code and complex projects (i.e. ~1M lines)?
All the similar questions I found talked about static analysis which we all ready do.
Thank you!
You might want to look at the code coverage tools that are used in testing. The idea of these tools is that they instrument the code and after running the set of tests you know what lines of code were executed at least once and what lines were never executed. After that you can improve tests.
The same thing can be used to identify dead code in case if you have diverse enough execution environment.
I don't know what platform you are on but we have used Gcov with success if you're compiling with the gnu toolchain:
http://gcc.gnu.org/onlinedocs/gcc/Gcov.html
I'm using Visual Studio 2010, and in my C++/CLI project there are two Code Analysis settings:
Enable Code Analysis on Build
Enable Code Analysis for C/C++ on Build
My question is about the second setting.
I've enabled it and it takes a long time to run and it doesn't find much.
Do you recommend enabling this feature? Why?
The two options you specify control the automatic execution of Code Analysis on managed and native C++ respectively.
Code Analysis of managed code is performed by FXCop engine analyzing the generated IL.
Code Analysis of native code is performed during compilation by the PREFast engine analyzing the C++ source code.
I strongly encourage you to require your developers to have run CA on their code before checking it in. If you don't, you're:
Delaying the process of ensuring that your code has no known vulnerabilities and issues that could otherwise have been systematically removed from your product's source.
Denying your developers their right to improve their skills by learning incrementally what code they should not be writing and why.
Selling your customers short because they're the ones who will suffer from crashes and security issues when they're using your product.
Further, if you're writing native C++ and have not already planned to start adorning your code with SAL Annotations, then, frankly, someone at your place of work deserves to be dragged out into the street and humiliated! There's some great stuff coming down the pipe shortly in the next version of the SAL annotations - get on it now and be way ahead of the curve compared to your competitors! :)
Never did anything for me. In theory, it's supposed to help catch logical errors, but I've never found it to report anything.
We are using LINT to do a static code analysis for plain C++ applications (no .Net, no C++/CLI).
This is different from what you are using but probably the same principles can be applied.
We execute LINT like this:
During a build, only the changed sources (CPP files) are run through LINT. Possibly many more files are being recompiled (if a header file is changed), but only the changed .CPP files are run through LINT.
Run the static code analysis on all files on a Continuous Integration server. If it finds something, let it mail the error to the developers that most recently committed changes to the versioning system, or to the main developer.
What you could do additionally is to perform a static code analysis on all files that are committed to your versioning system. E.g. in Subversion you could do this in a commit-trigger.
Is anyone aware of extensions to CppUnit that can be used to make assertions on a test by test basis concerning memory leaks.
i.e. CPPUNIT_ASSERT_NO_LEAKS()?
Essentially, I want to be able to fail specific tests when the execution of the test results in leaked memory.
If you're running on Linux, you could run your tests with memcheck.
The Client Requests section of the manual describes several useful macros, of which one is noted as being useful for testing:
VALGRIND_COUNT_LEAKS: fills in the four arguments with the number of bytes of memory found by the previous leak check to be leaked, dubious, reachable and suppressed. Again, useful in test harness code, after calling VALGRIND_DO_LEAK_CHECK.
The macro is defined in memcheck.h (likely in /usr/include/valgrind), and the sequence you want will resemble
unsigned long base_definite, base_dubious, base_reachable, base_suppressed;
VALGRIND_DO_LEAK_CHECK;
VALGRIND_COUNT_LEAKS(base_definite, base_dubious, base_reachable, base_suppressed);
// maybe assert that they're zero!
// call test
unsigned long leaked, dubious, reachable, suppressed;
VALGRIND_DO_LEAK_CHECK;
VALGRIND_COUNT_LEAKS(leaked, dubious, reachable, suppressed);
CPPUNIT_ASSERT_EQUAL(base_leaked, leaked);
// etc.
Repeating that for every test would be a pain, so you might write macros of your own or, even better, a specialized TestRunner.
CPPUNIT doesn't have memory leak checks support by default.
The project has been recontinued now (it was stopped for a long time) and this may be a feature of CPPUNIT2, you can propose (or write) to the authors.
If you're looking for a Unit test framework with memory leak detection support you can try looking at CppUTest. It is the project used by Martin Fowler and Bob Martin on some TDD courses. It is pretty good.
On Windows it would be a pretty simple matter of using some calls to the debug heap to get CppUnit to act on this information using _CrtMemCheckpoint() and _CrtMemDifference():
http://msdn.microsoft.com/en-us/library/wc28wkas.aspx
There are drawbacks:
you'd have to place something manually at the start of the test to get the checkpoint (maybe there's a way to integrate that into CppUnit somehow)
it's Windows only (there's probably something similar on the various other platforms)
it'll only work for builds with the Debug CRT
Where I work we build our unit tests with purify. Then our continuous integration platform pulls both the number of testcases that succeeded/failed and the number of leaked bytes (+ lint and coverity results) and shows it on a web page. I can highly recommend doing it this way.
Sorry for not providing the solution you wanted.
I know it's a little too late to answer to this question. But here is a great tool from Microsoft. I am a linux user now but I have used this when I was writing codes in windows (Visual C++ and Qt)
http://www.microsoft.com/en-us/download/details.aspx?id=20028
No idea of that, but you could use something like the Fluid Studios Memory Manager code and hook that in yourself with some tweaking. Either that or compile it into your test application and then have a script that runs the application once for each test and collate the memory tracking results.
Run your unit tests with valgrind. The unit test framework which I use allows you to run one or more individual unit tests so you can detect which one is causing the leak.
I know it's not CppUnit, but boost::test can do memory leak detection.
From http://www.boost.org/doc/libs/1_39_0/libs/test/doc/html/execution-monitor/user-guide.html:
void detect_memory_leaks( bool on_off );
void break_memory_alloc( long mem_alloc_order_num );
Background
I have an application written in native C++ over the course of several years that is around 60 KLOC. There are many many functions and classes that are dead (probably 10-15% like the similar Unix based question below asked). We recently began doing unit testing on all new code and applying it to modified code whenever possible. However, I would make a SWAG that we have less than 5% test coverage at the present moment.
Assumptions/Constraints
The method and/or tools must support:
Native (i.e. unmanaged) C++
Windows XP
Visual Studio 2005
Must not require user supplied test cases for coverage. (e.g. can't depend on unit tests to generate code coverage)
If the methods support more than these requirements, then great.
NOTE: We currently use the Professional edition of Visual Studio 2005, not the Team System. Therefore, using Team System might be a valid suggestion (I don't know, I've never used it) however I'm hoping it is not the only solution.
Why using unit tests for code coverage is problematic
I believe that it is impossible for a generic tool to find all the dead (e.g. unreachable code) in any arbitrary application with zero false positives (I think this would be equivalent to the Halting problem). However, I also believe it is possible for a generic tool to find many types of dead code that are highly probable to in fact be dead, like classes or functions which are never reference in the code by anything else.
By using unit tests to provide this coverage, you no longer using a generic algorithm and are thus increasing both the percentage of dead code you can detect and the probability that any hits are not false positives. Conversely, using unit tests could result in false negatives since the unit tests themselves might be the only thing exercising a given piece of code. Ideally, I would have regression testing that exercises all externally available methods, APIs, user controls, etc. which would serve as a baseline measurement of code coverage analysis to rule out certain methods from being false positives. Sadly however, I do not have this automated testing at the present time.
Since I have such a large code base with such a low test case coverage percentage however, I'm looking for something that could help without requiring huge amounts of time invested in writing test cases.
Question
How do you go about detecting dead code in an automated or semi-automated fashion in a native C++ application on the Windows platform with the Visual Studio 2005 development environment?
See Also
Dead code detection in legacy C/C++ project
I want tell the VC++ Compiler to compile all code. Can it be done?
Ask the linker to remove unreferenced objects (/OPT:REF). If you use function-level linking, and verbose linker output, the linker output will list every function it can prove is unused. This list may be far from complete, but you already have the tools needed.
We use Bullseye, and I can recommend it. It doesn't need to be run from a unit test environment, although that's what we do.
Use a code coverage tool against your unit test suite.