Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
I use the Boost Test framework for my C++ code but there are two problems with it that are probably common to all C++ test frameworks:
There is no way to create automatic test stubs (by extracting public functions from selected classes for example).
You cannot run a single test - you have to run the entire 'suite' of tests (unless you create lots of different test projects I guess).
Does anyone know of a better testing framework or am I forever to be jealous of the test tools available to Java/.NET developers?
I've just pushed my own framework, CATCH, out there. It's still under development but I believe it already surpasses most other frameworks.
Different people have different criteria but I've tried to cover most ground without too many trade-offs.
Take a look at my linked blog entry for a taster. My top five features are:
Header only
Auto registration of function and method based tests
Decomposes standard C++ expressions into LHS and RHS (so you don't need a whole family of assert macros).
Support for nested sections within a function based fixture
Name tests using natural language - function/ method names are generated
It doesn't do generation of stubs - but that's a fairly specialised area. I think Isolator++ is the first tool to truly pull that off. Note that Mocking/ stubbing frameworks are usually independent of unit testing frameworks. CATCH works particularly well with mock objects as test state is not passed around by context.
It also has Objective-C bindings.
[update]
Just happened back across this answer of mine from a few years ago. Thanks for all the great comments!
Obviously Catch has developed on a lot in that time. It now has support for BDD style testing (given/ when/ then), tags, now in a single header, and loads of internal improvements and refinements (e.g. richer command line, clear and expressive output etc). A more up-to-date blog post is here.
Take a look at the Google C++ Testing Framework.
It's used by Google for all of their in-house C++ projects, so it must be pretty good.
http://googletesting.blogspot.com/2008/07/announcing-new-google-c-testing.html
http://code.google.com/p/googletest
Boost.Test does allow to run test case by name. Or test suite. Or several of them.
Boost.Test does NOT insists on implementing main, though it does make it easy to do so.
Boost.Test is NOT necessary to use as a library. It has single header variant.
I just responded to a very similar question. I ended up using Noel Llopis' UnitTest++. I liked it more than boost::test because it didn't insist on implementing the main program of the test harness with a macro - it can plug into whatever executable you create. It does suffer from the same encumbrance of boost::test in that it requires a library to be linked in. I've used CxxTest, and it does come closer than anything else in C++-land to automatically generating tests (though it requires Perl to be part of your build system to do this). C++ just does not provide the reflection hooks that the .NET languages and Java do. The MsTest tools in Visual Studio Team System - Developer's Edition will auto-generate test stubs of unmanaged C++, but the methods have to be exported from a DLL to do this, so it does not work with static libraries. Other test frameworks in the .NET world may have this ability too, but I'm not familiar with any of those. So right now we use UnitTest++ for unmanaged C++ and I'm currently deciding between MsTest and NUnit for the managed libraries.
I'm a big fan of UnitTest++, it's very lightweight, but does the job. You can run single tests there easily.
Great question! A few years ago I looked around forever for something worth using and came up short. I was looking for something that was very lightweight and did not require me to link in some libraries... you know something I could get up and running in minutes.
However, I persisted and ended up running across cxxtest.
From the website:
Doesn't require RTTI
Doesn't require member template functions
Doesn't require exception handling
Doesn't require any external libraries (including memory management, file/console I/O, graphics libraries)
Is distributed entirely as a set of header files (and a python script).
Wow... super simple! Include a header file, derive from the Test class and you're off and running. We've been using this for the past four years and I've still yet to find anything that I'm more pleased with.
Try WinUnit. It sounds excellent, and is recommended by John Robbins.
I like the Boost unit test framework, principally because it is very lightweight.
I never heard of a unit-test framework that would generate stubs. I am generally quite unconvinced by code generation, if only because it gets obsolete very quickly. Maybe it becomes useful when you have a large number of classes?
A proponent of Test Driven Development would probably say that it is fundamental that you run the whole test suite every time, as to make sure that you have not introduced a regression. If running all the tests take too much time, maybe your tests are too big, or make too many calls to CPU intensive functions that should be mocked out? If it remains a problem, a thin wrapper around the boost unit-tests should allow you to pick your tests, and would probably be quicker than learn another framework and port all your tests.
http://groups.google.com/group/googletestframework, but it's pretty new
I'm using tut-framework
Aeryn is another framework worth looking at
Visual Studio has a built-in unit testing framework, this is a great link to setting up a test project for win32 console application:
http://codeketchup.blogspot.ie/2012/12/unit-test-for-unmanaged-c-in-visual.html
If you are working on a static DLL project it is much easier to set up as other have pointed out external tesing frameworks like GTest and Boost have extra features.
CppUnit was the original homage to JUnit.
Eclipse/JUnit is a solid package for java, and there are C++ extensions/equivalents for both. It can do what you're talking about. Of course, you'd have to change IDEs...
I too am a fan of UnitTest++.
The snag is that the source distribution contains almost 40 seperate files. This is absurd. Managing the source control and build tasks for a simple project is dominated by looking after all these unit testing files.
I have modified UnitTest++ so that it can be integrated with a project by adding one .h and .cpp file. This I have dubbed "cutest". Details are at http://ravenspoint.com/blog/index.php?entry=entry080704-063557
It does not automatically generate test stubs, as asked for in the original question. I cannot help thinking that such a feature would be more trouble than it is worth, generating vast amounts of useless code "testing" accessor functions.
I would imagine automatically stubbing out test functions would be more of a function (of scripts for the framework or) the development environment in question. Supposedly CodeGear's C++Builder applications will quickly generate test code for user functions.
Andrew Marlow's Fructose library's worth checking out... http://fructose.sourceforge.net/
I recall his documents containing a fairly detailed analysis and comparison of other offering at the time he wrote Fructose, but can't find a URL direct to that document.
I'm trying out Igloo, also a header only C++ test suite, even it's two included dependencies are header only.
So, it's pretty straightforward and simple. Besides the included example on github, there's examples and more details at the main site, igloo-testing.org. I'll update this later as I get more experience with it and other frameworks.
Related
I want to know if anyone of you guys use TDD in your c++ projects and how it performs compared to managed languages like C# and Java.
And what frameworks you guys are using to automate tests on c++ projects?
Two useful C++ test frameworks that don't seem to have been mentioned yet are Boost test and Google Test.
Test Driven Development is possible in any language. You need the right testing tools and methodologies for the language, and may possibly need a custom testing infrastructure for your project.
I have found CppUnit (at least 1.x) to be a very poor framework -- it seems to use Java/C# idioms in a C++ language and does not have support for STL constructs.
If you want a good example of Test Driven Development (in C), look at the Wine project -- http://test.winehq.org/data/ shows their test results across the different versions of Windows, Wine and the different commits into the Wine repository. They have their own custom test infrastructure.
I recently moved from a C# project that was developed using TDD to a project that is using C++. I was dreading it quite a bit, but I find that doing C++ with TDD is a lot more enjoyable and the code is more robust than I remember from past (non-TDD) experiences with C++.
We are using Google Test. It is not as easy to use as NUnit/MbUnit, but it seems to work pretty well. There is also a Google mocking framework http://code.google.com/p/googlemock , but I have not been using that yet.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
I'm a student and i want make a build tool as a side project for myself because none of the current build tools seem to attract me. There's ant, but i really dislike looking at XML (i don't know why, but i really don't see the appeal of tags; it puts me off visually and cannot be made neat in my eyes). There's maven, but i really don't want to be working on a project just to have it fail all of a sudden (this is based on the research I've done where people say that there are times that maven fails to build all of a sudden. This could just be total BS but i'd rather not find out the hard way), plus there's some XML involved. I really liked make, but it isn't portable, and even though the chances of me using a non-Windows PC are next to nil, I am unfortunately, a computer science student who has been properly trained to always assume the worst case.
Currently, i am looking into Gradle. I'm still trying to figure it out (i am a really slow learner) but i like the syntax. It might also be beneficial to note that i am so shallow that i don't really care about the back end stuff or any of the advanced features cause i don't really understand dependency management and stuff just quite yet. All i care about is that the configuration syntax or make file looks clean, that it works without requiring internet access, absolute paths, or anything of the sort, and that it works consistently and doesn't take too long. I believe that the syntax is important because, like all code, if it looks ugly, you won't want to read it.
I want to make a build tool that is simple, functional, and portable (i'm gonna be running it off my External HDD with Java in it). I'd really appreciate any suggestions this community has to offer, such as "It really is better to just stick with ant or maven cause they work already" or "be sure to avoid make's dependency issue". This will just be a side project so i can work on my Java and maybe learn something new so even if i fail, i still might learn something new so no need to comment on that.
If you haven't already tried it, try Automated Build Studio. We have used if for a couple of years now and it works fine for us.
Another existing tool you might choose to ignore is SCons. It's written in Python and has some very nice features including, possibly relevant to you, easy extensibility.
And yes, I think you are going to crash and burn. Far better to learn one of the existing tools properly than to reinvent this particular wheel. Use your creative urges and time to write something truly new.
Furthermore, and since you are a student allow me to stand on my soap-box: one of the lessons that new entrants to the IT profession have to learn is to use the tools you are given, not to expect to be allowed the time and money to develop a new one. Don't expect to be able to rock up to work on day one and try to make a case for writing 'MyMake' because GNU Make (or whatever) doesn't float your boat.
Here's just an idea:
ant is really not such a bad tool. Maybe you could consider building a front end for it that translates a DSL (Domain Specific Language) of your own design into ant's XML, and hide the XML from your tool's users?
You may also want to take a look at Rake. Someone recently ranted in his blog about how terrible Maven is (and the debate goes on) and he much, much prefers Rake.
I started one, some 7 years ago. I am still using it. No one else uses it. http://gna.org/projects/maker/
I started on some same ground than you: multi platform (was Linux, DEC Alpha, and Windows at the time) with a new shiny lanquage: Python. I was getting ideas from Miller's Recursive Make Considered Harmful in the sense that I had a source file for each executable (same for shared library, or static library) that was listing the C/C++ source files, and dependencies on other libraries. The main feature was to generate on the fly a makefile that is fed into gmake (that does a perfect job, provided it has a perfect makefile) to build all the binaries in one call, managing all the dependencies (with gcc -MD options).
Over time. it evolved mostly into a tool that can use Visual Studio C++ project and solution files to compile on Linux. And I am struggling to keep it up-to-date with new versions of Visual, and new values of project parameters and project properties that my coworkers use.
I wouldn't recommend starting a new tool.
Why would using Maven mean that your build fails "all of a sudden"?
I would advocate always using continuous integration (e.g. cruise control for ant or Hudson for maven) regardless of the build tool you use. This should eliminate a build failing "all of a sudden"
There's maven, but i really don't want to be working on a project just to have it fail all of a sudden.
What does that mean?
Anyway, if you're into Groovy, I recommend Gant.
Cobbling together your own build tool doesn't sound like a great use of your time, especially when established tools such as ant and maven exist and have such vast user bases that are finding (and fixing) bugs with those tools.
I don't understand your comment about maven causing build failure all of a sudden.
XML will haunt you everywhere you go. Invest in an excellent XML visualizer / editor, and go to town with Ant anyhow!
what would be (rough estimation, average, of course) the initial learning and setup curve and subsequent overhead for using Maven for C++/Eclipse/Linux project of small to medium size?
We are 4 developers at the beginning of the way. We currently have ~20 native eclipse C++ (CDT) "projects", which we compile interactively. We would like to have an automated checkout & build script.
It seems a bit overkill at this stage, but perhaps we should adopt it sooner then later, provided that it does not incur an overhead. We don't have bandwidth for extensive configuration management right now. Thanks a lot!
EDITED / DETAILED:
I realize I haven't described my needs well enough. Having read the references provided below, I see that CI tool seems an overkill for us at the moment. What I'd like to have is a build tool that is well integrated with eclipse on one hand, and allows offline, non-interactive builds on the other. I enjoy the simplicity of working with eclipse projects: you just add files, add references to internal components and 3rd part libs as they add up, and that's it. You don't need to manually maintain makefiles or the like. The trouble with it, as with MSVS a few years ago when I worked with it, is that it does not give you an option of non-interactive builds. So, does such tool exist?
First, while Maven has some support to build C++ projects with the maven-native-plugin or, if you already are using Make, with the maven-make-plugin from the c-builds suite, this is not a common use case and there aren't widely used. So while it should be possible, you won't get support and find resources easily (just Google a bit or browse the maven users list to get an idea).
Second, if you add to this that you'll have to learn Maven in the same time, then it seems reasonable to say that you are not taking the easiest path.
So, instead, I'd stick with more traditional tools and/or Ant. For the continuous integration itself, I've seen several references mentioning the use of CruseControl to build a C++ project. Refer to What continuous integration tool is best for a C++ project? or UsingCruiseControlWithCplusPlus for example. But I guess the principles are transposable to another CI engine (like Hudson that I find much more easy to use than CruiseControl).
How effective is the CXX test framework, given that you are writing unit test cases around the code that you have written. Any bug in the code might as well get translated into a bug in the unit test code as well? Isn't it something like two negatives make a positive?
Also, the time and effort spent on CXX is at least equal to if not more than the coding effort.
Need your thoughts on this since I'm not in favor of this framework being used in my project and am looking for strong points to oppose it.
On the other hand, if you think it's beneficial, please do enlighten me :).
Google offers a fantastic C++ testing framework that I have used... I never used any other C++ testing framework, and had limited experience with Junit, and I was able to pick this up very quickly, as the documentation is good. It's important to use a good testing framework, because testing is too important to give up on because of frustration with the framework. Here is a link:
http://code.google.com/p/googletest/
Hope this helps!
CXX is not very active, and writing unit test generally involves a lot of efforts. But when the first refactoring comes in, you're very grateful of the spent effort.
I've used Boost.Test & CPPUNIT. I would prefer a little bit Boost.Test, but yes, you have to write your own projects, files etc.
If you know a tool to generate your skeleton from your code, I'm all ears. :)
I would suggest that you give a try to Boost.Test and CPPUNIT. If you think there are better it will give you good rounds to oppose CXXUNIT as you will propose alternatives.
I am using cxxtest. Regressive testing is an expensive task that we only use to validate our software libraries - which provide a platform independent layer for our apps. This is to ensure that all changes will not affect the stability of the code since so many apps and projects and dependent on them.
We couple cxxtest with coverage analysis to ensure that test coverage is sufficient and also with CruiseControl to automate it.
But we do not do this for apps. Too much effort.
Building a test app is just as difficult as writing the whole library itself. I agree that you will need to work out whether it is worth your while.
I think Joel has something to say about this too:
http://www.joelonsoftware.com/items/2009/01/31.html
I prefer header-only test frameworks, here are two of them: TUT and Catch . I used TUT before in several projects, and found Catch not long ago.
1) TUT -- C++ Template Unit Test Framework
TUT is a small and portable unit test framework for C++.
TUT is very portable, no matter what compiler or OS you use.
TUT consists of header files only. No libraries required, deployment has never been easier.
Custom reporter interface allows to integrate TUT with virtually any IDE or tool in the world.
Support for multi-process testing (testing deadlocks and timeouts is under way).
TUT is free and distributed under a BSD-like license.
Tests are organised into named test groups.
Regression (all tests in the application), one-group or one-test execution.
Pure C++, no macros!
2) Catch -- A modern, C++-native, header-only, framework for unit-tests, TDD and BDD
What's the Catch?
Catch stands for C++ Automated Test Cases in Headers and is a
multi-paradigm automated test framework for C++ and Objective-C (and,
maybe, C). It is implemented entirely in a set of header files, but is
packaged up as a single header for extra convenience.
For C++, there are lots of good unit test frameworks out there, but I couldn't find a good one for functional testing. With functional testing, I mean stuff which touches the disk, requires the whole application to be in place etc.
Point in case: What framework helps with testing things like whether your I/O works? I've got a hand-rolled system in place, which creates temporary folders, copies around a bunch of data, so the tests are always done in the same environment, but before I spend more time on my custom framework -- is there a good one out there already?
I wrote one from scratch three times already - twice for testing C++ apps that talked to exchanges using FIX protocol, once for a GUI app.
The problem is, you need to emulate the outside world to do proper system testing. I don't mean "outside of your code" - outside of your application. This involves emulating end users, outside entities, the Internet and so on.
I usually use perl to write my system testing framework and tests, mostly because it's good with accessing all sorts of OS facilities and regexps are first-class citizens.
Some tips: make sure your logs are easy to parse, detailed but not too verbose. Have a sane default configuration. Make it easy to "reset" the application - you need to do it after each test.
The approach I usually use is to have some sort of "adapter" that turns the application's communications with the outside world into stdin/stdout of some executable. Then I build a perl framework on top of that, and then the test cases use the framework.
Below I list a couple of tools and larger testing applications of which I am aware. If you provide more information on your platform (OS, etc.) we can probably provide better answers.
For part of what you require, Microsoft provides the Application Verifier:
Application Verifier (AppVerifier) is a runtime verification tool used in testing applications for compatibility with Microsoft Windows XP. This tool can be used to test for a wide variety of known compatibility issues while the application is running. This article details the steps for using AppVerifier as an effective addition to the application development and testing cycles.
Application Verifier can be useful for testing out low memory conditions, other low resources, and other API usage.
Another part of the puzzle, is the Microsoft Detours package, which can be used to replace API calls with your own code (useful for say, returning error codes for tests that are hard to set up).
Detours is a library for instrumenting arbitrary Win32 functions on x86, x64, and IA64 machines. Detours intercepts Win32 functions by re-writing the in-memory code for target functions. The Detours package also contains utilities to attach arbitrary DLLs and data segments (called payloads) to any Win32 binary.
There are other, larger (and more expensive) comprehensive packages available too. Borland makes Silk.
Automated Software makes TestComplete. The selection of one of these tools would be up to your needs for your applications.
IBM/Rational provides the Rational Functional Tester, which is available across many platforms, and feature-rich.
Hi I am not sure if the framework we have helps in your situation but it hooks into Rational Functional Tester and allows the user to create various datasets to be attached to different tests and to change the enviornments without changing the scripting and reuses the automation in an efficient way.
Have a look if your interested:
http://www.testpro.com.au/Test-Automation-Framework.html