I'm in the middle of implementing a unit test infrastructure for a large C++ project, and due to political reasons I'm almost sure CppUnit will be pushed as the unit testing framework.
I'm trying to identify mock frameworks that blend with CppUnit. I've found mockpp, and I've heard that Google Mock should work.
What frameworks work alongside CppUnit?
Mocking libraries are typically independent of the unit testing framework. They accomplish two different jobs, and frankly don't have much reason to talk to each other. Where they do integrate is in answering these questions:
When should I create my mock objects?
When should I initialize the mock objects with my expectations?
When should I validate the mocks were called as expected?
And you do that at the appropriate points in your tests.
For one example, check opmock. http://sourceforge.net/projects/opmock/
According to their wiki, opmock is easily called from a CppUnit test. See http://sourceforge.net/p/opmock/wiki/Using%20Opmock%20with%20other%20unit%20testing%20frameworks/
Related
I don't mean defer all unit-testing until an integration test passes. The unit tests I'm referring to are those that verify that the SUT interacts with the 3rd-party mystery API correctly.
The rationale for deferring these unit tests is that they verify something that is unknown. If I don't know how the 3rd-party mystery API works, how can I even write a unit test to ensure that the SUT uses the 3rd-party API correctly? Only once some minimal integration tests pass do I actually know what behavior to verify.
Of course, all other unit tests should be written before the SUT is written in the usual way (red, green, refactor).
Under the assumption that it's an external api I'd argure that your tests probably wouldn't qualify as a unit test anyway. They'd be integration tests.
How about writing a mock to test your code as is and then writing proper integration tests when you can access the 3rd party api? Also, if you're unit testing third party stuff, then your build will fail as soon as those things are unavailable which might be a problem in the long run.
A few years later, I have done much more unit testing, and feel I understand this better. Unit tests are good at verifying that a piece of code is correct ASSUMING that you correctly understand how to use the API's that are faked (mocked / stubbed). But to fake out those API's correctly, you must first understand how they work. So after reading the documentation for those API's, you can check your understanding by doing a proof-of-concept--even if that means testing code manually after writing it--or by writing an integration test. Once you are semi-comfortable with your understanding of the thrid-party API, you can fake it in a unit test and follow TDD once again.
Wrappers for third-party API's are an excellent thing to write integration tests for, where you merely test your minimal wrapper without faking the third-party API to ensure you correctly understand it. Some of these tests might not really be tests, but demonstrations written like tests. They come in handy later, too, because you can review them if you forget how something works, help a coworker understand the API, and discover subtle breaking changes when you upgrade the API.
Today, I often write integration tests of wrappers, write demonstrations / examples of third-party API's exercising corner cases relevant to my situation, and even write production code and manually test it if I have to. I only have to when there is an ambiguity or unclarity in the API I'm using. Once I feel confident about how to use it, it's time for "normal" TDD.
For those of us who like to learn by reading good code, what are some of the projects you've seen which provide a great example of a medium to large suite of unit tests in action?
This isn't a question about what your favourite unit testing framework is, but it could be helpful to add which unit testing and/or mocking frameworks are used by the projects you mention.
Any platform will do, but I'm mainly interested in projects which use an xUnit-style unit testing framework in .NET.
Apache CXF has a mind-numbingly large array of unit tests. They are written to Junit, and include the JUnit-Spring mechanisms as well as one of the mock libraries. They include launching processes and many other mechanisms you might need some day.
For example, ReportLab uses Python's unittest module (also known as PyUnit) and has lots unit test that you might want to take a look at.
I'm also using unittest myself. It is very similar to Java's jUnit framework and thus very handy for writing tests in no time.
The Service Factory has about 780 unit tests, including some that test code meant to run inside of Visual Studio. It has some very good examples of mocking such code.
The Enterprise Library has even more.
NunitLite has a unit tests suite but I'm not sure it qualifies as a medium-sized project. I seem to remember NUnit also has unit tests, but its site currently is unavailable.
I've introduced visitors as one of core architecture ideas in one of my apps. I have several visitors that operate on a same stuff. Now, how should I test it? Some tests I'm thinking of are a bit larger then a unit test should be (integration test? whatever) but I still wanna do it. How would you test code like the C++ sample from wiki art on Visitor Pattern
Unit testing isn't really about testing patterns, it is about testing the correct implementation of methods and functions. The visitor pattern is a specific class structure, and for each of the classes (ConcreteVisitor and ConcreteElement, specifically) involved you'll want unit tests.
When you've developed confidence that your class methods are behaving OK, you could use your unit test framework to develop integration tests as well. Do not start integration testing rightaway: you'll find that you develop a lot of integration tests that are actually testing the behavior of a specific class, i.e. unit tests.
Whether you need mock objects or can use 'real' objects is a different matter. This depends a lot on whether the objects behave nice enough for unit test purposes (i.e. they do not pull in a lot of additional dependencies etc.), and whether the objects themselves are unit tested (i.e. you need to be able to trust these objects 100%). The mock vs. real objects issue has been addressed on stackflow before, so search the unittest tags.
make a test visitor object and make it visit things.... test that it visited the right things.
You can create mock objects and have your visitors visit them, and then create mock visitors, and test that the right actions were performed.
Recently there has been quite some hype around all the different mocking frameworks in the .NET world. I still haven't quite grasped what is so great about them. It doesn't seem to be to hard to write the mocking objects I need myself. Especially with the help of Visual Studio I quickly can write a class that implements the interface I want to mock (it auto-generates almost everything for me) and then write an implementation for the method(s) I need for my test. Done! Why going through the hassle of understanding a mocking framework for the sole purpose of saving a few lines of code. Or is a mocking framework not only about saving lines of code?
Once I finally got the hang of mock objects, I realized that they're essential for unit testing for the same reason that double blind testing or control groups are essential for scientific trials: they isolate what you're actually testing.
If you're testing a class which has quite a bit of interaction via other interfaces, you not only save the lines of code on having to mock each and every interface, but you also gain the ability to do things like "throw an exception if an unexpected method is called" or "exception if these methods are called out of order". You can get remarkably sophisticated with mock frameworks, and though I'll quickly admit there's a large learning curve, when you get up to speed they'll help make your unit tests more thorough without being bloated.
You actually identified one of the key points of a mock framework in your question. The fact that you code the mocks yourself is not something the developer should be concerned with. The mocking frameworks give you implementations of interfaces programatically, plus they are functional (based on your setup of the mock).
What do you do if you are testing an ICustomerDAO, for example, and you want to test some method 14 times each with different outcomes? Implement 14 different classes manually? I doubt that anyone would want to do that.
Mocks give you the power to define what will happen with parts of your classes when you are not concerned with whether or not they will actually work, like throwing exceptions whenever you want them to, returning zero results and making sure you handle that correctly, etc...
They are a great unit testing tool.
Previous questions that may help:
What is a mock and when should you use it?
Mockist vs classical TDD
I find that using a mocking framework allows me to generate tests a lot faster and with better verification that what I expect to happen in the test actually is happengin. I have in the past implemented stubs or fakes myself. I found that I needed to generate stubs specific to the test that I wanted and this took a lot of time. I can create the same test much faster using a mocking framework. The good ones support the generation of fakes, stubs or mocks with straightforward syntax.
It takes a while to get the hang of it, I avoided it for a while but now wouldn't try to work without a mocking framework for the reasons #Chamelaeon states.
Roy Osherove had a poll about Mock Frameworks and down in the comment section, there is a discussion (albeit brief) about whether one needs a Mock Framework or not.
I personally have been manually doing exactly as you stated and it has worked well enough, but this has mainly been out of habit rather than a closely-held opinion on mock frameworks in general.
Well I certainly don't think that you NEED a mocking framework. It's a framework like any other, and it's ultimately designed to save you some time and effort. You can also do things like roll your own common data structures like stacks and queues, but isn't it generally easier to just use the ones built into the class libraries that ship with the compiler/IDE of your language of choice?
I'm sure there are other compelling reasons for using mocking frameworks, though I'd leave it to the TDD and unit testing gurus to answer.
For the same reason you wouldn't try to write unit tests without NUnit. A mocking framework will assist you in verifying state and behavior over hundreds of unit tests. It's worth the 2 weeks or so of pain to get up to speed and really helps you focus on what needs to be tested.
One thing that troubles me about a mocking framework is that "what a function should o/p given an i/p" via
when(mock.someMethod("some arg")).thenReturn("something");
statement is spread across many unit test classes.
Let me elaborate with an example. Lets say there was a DAO Interface function getEmp(int EmpID) which was returning an Employee Object when passed an Employee ID as a parameter. Assume that this function was being mocked by 10 different unit test classes. Now if in the future, this function were changed to return a newer version of the Employee Object, one would have to go to each of the 10 different classes to update this change.
The disadvantages are as follows...
a) I don't know how to figure out all the classes which mock this function so that I can go update this change.
b) My existing test cases which consumes the mock DAO object continue to be blissfully unaware of the changes that have happened to the DAO Interface because the mock has not changed and hence continue to be green.
Ideally, if I were to have coded a single mock class myself and consumed it everywhere, I would have just one place to update for the newer version of the Employee object. Also, once I update at this one place, all my existing test cases which consume the mock would break and I would then know exactly what places I need to go and do an update for the new Employee Object.
Any thoughts on my views..
One of the good things about a mocking framework is that it allows setting expectations on the objects being mocked. With the expectations I can then set up all sorts of conditions to exercise the code thats being tested.
An isolation framework or mocking framework allows you to test the code you want, without its dependencies. It makes for short running tests, allows you to debug quickly, and easily build a safety net of tests around the code. Different frameworks have different features, and as said before - it's a tool, and you should select the right tool for the job.
I've use rhino mocks for a mocking framework. I and 5 other developers used it on a large enterprise application that was an 8 month project. We used tdd on the project. Was it worth it? I guess. Was there such a massive huge selling point to using mocks that I have to use it on every project? In my opinion, no. It is not something that is necessary, it is just a tool that you can use if you want to try it out. Some projects you can roll out your own mock classes as some here say they prefer - it is easier. Other projects are larger and may require a mocking framework. The key word (in my opinion) is MAY require... how much code coverage do you require? To me, that is another consideration to using mocks. The project I did with tdd/rhino mocks we were required to have 80% code coverage so the mocks helped us attain that. If our code coverage requirements were less, for example 40%, we probably would have not used a mocking framework and just wrote our own mock classes as others mention they do.
I am curious about how many of you folks incorporate mocking of objects (frameworks like JMock, NMock, RhinoMocks hand in hand with unit testing frameworks) into your daily development process. What are your experiences?
You see, I develop on top of a GIS (geographic information systems) platform, in which most of work pertains to some way of handling data. As its data object model is quite complex (many many classes and interfaces, all COM-based), it is also quite difficult and cumbersome to mock. In this case, mocking incurs a great deal of overhead when writing test suites. I wonder if there are people in similar situation, or just, how does mocking (in whatever situation you are in) work for you.
On a recent project that I worked on we used mock objects extensively in our unit testing approach. The project was 100% Java and moderately sized (about 100,000 lines of non-commented code). It was a Swing-based desktop application - and the only effective way that we found to test the user interface logic was through an MVC variant design that allowed us to use mock objects to substitute for the actual Swing user interface classes for the automated testing. We also used mocking extensively in the testing of our data access layer (Hibernate/DAOs).
In the user inteface use, the Mocks were easy and straightforward to build. And the design of the application (Fowler Passive View) easily incorporated mocks. This was not the case for the mocks used in testing the data access layer. But I can say that it was clearly worth the effort. In fact, most of the 'effort' really focused on coming up with a reusable solution that minimized the work that a developer had to do to create each individual mock. I'd recommend taking the time to dig into and discover an approach for your situation that allows you to easily mock up your GIS data layer. That - or just manually mock up each class. Either way the ability to run the automated unit tests that rely on the mocks is worthwhile...
In my situation mocks work really nice. But I'm using Python, which is so dynamic that it makes many things involving testing much, much easier.
In situation like yours, when application is mainly data-driven (as far as I see), mocks may not be as useful. Just passing data in and watching it come out should be enough for testing. I would just make sure that application is modularized enough, so this approach can be applied to reasonably small components.
Mocking can be useful in some kind of project. But, sometimes mocking is very time consuming and the ROI of it is low.
Trying to test Sharepoint it seems that mocking is the only way, and only typemock will let you mock sealed classes.
Mocking is used very extensively in my case. Mocks are usually for the classes that has external dependencies, e.g. network, database, filesystem. Any of these can introduce flakiness in the tests if mocks are not used.
If the mocks that you find costly to write because there are a lot of fake data to populate, you could set some pre-populated data objects as constants and use them or slightly modified copies in your test. If such data objects has external dependencies, then maybe refactor it in a way you can separate the two concerns.
There is an initiative started by Dave Bouman to try and build a community library of Mocks for use in ArcObjects related unit testing. His blog and this svn repository have great information related to unit testing GIS systems
http://blog.davebouwman.net/CategoryView,category,Unit%2BTesting.aspx
http://svn2.assembla.com/svn/arcdeveloper/TestingUtilities/trunk/