I am curious about how many of you folks incorporate mocking of objects (frameworks like JMock, NMock, RhinoMocks hand in hand with unit testing frameworks) into your daily development process. What are your experiences?
You see, I develop on top of a GIS (geographic information systems) platform, in which most of work pertains to some way of handling data. As its data object model is quite complex (many many classes and interfaces, all COM-based), it is also quite difficult and cumbersome to mock. In this case, mocking incurs a great deal of overhead when writing test suites. I wonder if there are people in similar situation, or just, how does mocking (in whatever situation you are in) work for you.
On a recent project that I worked on we used mock objects extensively in our unit testing approach. The project was 100% Java and moderately sized (about 100,000 lines of non-commented code). It was a Swing-based desktop application - and the only effective way that we found to test the user interface logic was through an MVC variant design that allowed us to use mock objects to substitute for the actual Swing user interface classes for the automated testing. We also used mocking extensively in the testing of our data access layer (Hibernate/DAOs).
In the user inteface use, the Mocks were easy and straightforward to build. And the design of the application (Fowler Passive View) easily incorporated mocks. This was not the case for the mocks used in testing the data access layer. But I can say that it was clearly worth the effort. In fact, most of the 'effort' really focused on coming up with a reusable solution that minimized the work that a developer had to do to create each individual mock. I'd recommend taking the time to dig into and discover an approach for your situation that allows you to easily mock up your GIS data layer. That - or just manually mock up each class. Either way the ability to run the automated unit tests that rely on the mocks is worthwhile...
In my situation mocks work really nice. But I'm using Python, which is so dynamic that it makes many things involving testing much, much easier.
In situation like yours, when application is mainly data-driven (as far as I see), mocks may not be as useful. Just passing data in and watching it come out should be enough for testing. I would just make sure that application is modularized enough, so this approach can be applied to reasonably small components.
Mocking can be useful in some kind of project. But, sometimes mocking is very time consuming and the ROI of it is low.
Trying to test Sharepoint it seems that mocking is the only way, and only typemock will let you mock sealed classes.
Mocking is used very extensively in my case. Mocks are usually for the classes that has external dependencies, e.g. network, database, filesystem. Any of these can introduce flakiness in the tests if mocks are not used.
If the mocks that you find costly to write because there are a lot of fake data to populate, you could set some pre-populated data objects as constants and use them or slightly modified copies in your test. If such data objects has external dependencies, then maybe refactor it in a way you can separate the two concerns.
There is an initiative started by Dave Bouman to try and build a community library of Mocks for use in ArcObjects related unit testing. His blog and this svn repository have great information related to unit testing GIS systems
http://blog.davebouwman.net/CategoryView,category,Unit%2BTesting.aspx
http://svn2.assembla.com/svn/arcdeveloper/TestingUtilities/trunk/
Related
What is the advantage of mocking the container services in Unit testing of EJB 3.1?
The probable answers I get when I think about it are,
It improves the performance of the tests.
It does not abide by the rules of Unit testing as there is a lot of interactions with other APIs. (Please provide your views on this)
Other than these, do you think there are other advantages?
As many of you may know, it is possible to test some of the services provided by the container, like persistence, transaction management (eg. using Bitronix), messaging (eg. using Apache ActiveMQ and in-memory JNDI) out of the container in your own JVM. Still there is an argument that it is integration testing and unit testing should not be done that way.
According to me, if you can have a good performance in your tests, it is fine to use these third party implementations for unit testing because you do not have to spend too much time in mocking and mocking is heavily subject to developer errors. If a developer does not have a good understanding of mocking, he might end up mocking everything or in other words misusing mocking to turn the tests "green". Is this right? (Please provide your views on this)
After all, I never got any solid definition of unit testing :-). It depends on the author. Some define "unit" as the smallest unit that can be tested and some define "Depending on the context, these could be the individual subprograms or a larger component made of tightly related units."
Thanks.
If you have code which uses container services, then to test it, you will need to either mock those services, or use a real implementation. You have to do one or the other: without some implementation of the services, your code will not run, and so cannnot be tested.
Sometimes, you can refactor your code to remove the direct dependency on the container services, which will also remove the need to mock those services. But not always.
Mocking the container services provides more isolation than using a real implementation. It also gives you more control over and insight into the execution of your code. However, it also involves writing more code, and with it, more risk of introducing bugs (bugs in mocks, that can translate directly into bugs in application code).
There are some times when mocking definitely makes sense. For example, if you want to write a test that checks that your code is making the correct calls on UserTransaction, then that it much easier to do by mocking than by trying to instrument a real transaction monitor. If you want to write a test that checks that your code handles a particular SQLException correctly, then that is almost impossible to do without a mock.
Beyond those cases, as you point out, it is possible to write tests using the real services, or mocking them. As i think you have realised, the orthodox unit testing approach would be to mock them, or, in fact, to wrap them, and then mock the wrappers.
Whether this is actually necessary, or a good idea, is very much open to debate.
StackOverflow is not supposed to be for subjective questions, or for debates or discussions, so i hesitate to go into my opinion on this. Suffice to say that it is the same as i suspect yours to be - that the orthodox 'mock everything that moves' approach is unnecessary and harmful, and we would be much better off writing tests with less mocking, covering larger areas of real code. After all, real code is what we're going to ship to users, so why not test it?
I am starting to buy into BDD. Basically, as I understand it, you write scenario that describes well acceptance criteria for certain story. You start by simple tests, from the outside-in, using mocks in place of classes you do not implement as yet. As you progress, you should replace mocks with real classes. From Introduction to BDD:
At first, the fragments are
implemented using mocks to set an
account to be in credit or a card to
be valid. These form the starting
points for implementing behaviour. As
you implement the application, the
givens and outcomes are changed to use
the actual classes you have
implemented, so that by the time the
scenario is completed, they have
become proper end-to-end functional
tests.
My question is: When you finish implementing a scenario, should all classes you use be real, like in integration tests? For example, if you use DB, should your code write to a real (but lightweight in-memory) DB? In the end, should you have any mocks in your end-to-end tests?
Well, it depends :-) As I understand, the tests produced by BDD are still unit tests, so you should use mocks to eliminate dependency on external factors like DB.
In full fledged integration / functionality tests, however, you should obviously test against the whole production system, without any mocks.
Integration tests might contain stubs/mocks to fake the code/components outside the modules that you are integrating.
However, IMHO the end-to-end test should mean no stubs/mocks along the way but production code only. Or in other words - if mocks are present - it is not really end-to-end test.
Yes, by the time a scenario runs, ideally all your classes will be real. A scenario exercises the behaviour from a user's point of view, so the system should be as a user would see it.
In the early days of BDD we used to start with mocks in the scenarios. I don't bother with this any more, because it's a lot of work to keep mocking as you go down the levels. Instead I will sometimes do things like hard-code data or behaviour if it lets me get feedback from the stakeholders more quickly.
We still keep mocks in the unit tests though.
For things like databases, sure, you can use an in-memory DB or whatever helps you get feedback faster. At some point you should probably run your scenarios on a system that's as close to production as possible. If this is too slow, you might do it overnight instead of as part of your regular build cycle.
As for what you "should" do, writing the right code is far more tricky than writing the code right. I worry about using my scenarios to get feedback from the stakeholders and users before I worry about how close my environment is to a production environment. When you get to the point where changes are deployed every couple of weeks, sure, then you probably want more certainty that you're not introducing any bugs.
Good luck!
I agree with Peter and ratkok. I think you keep the mocks forever, so you always have unit tests.
Separately, it is appropriate to additionally have integration tests (no mocks, use a DB, etc. etc.).
You may even find in-betweens helpful at times (mock one piece of depended-on code (DOC), but not another).
I've only recently been looking into BDD and in particular jBehave. I work in fairly large enterprises with a lot of waterfall, ceremony orientated people. I'm looking at BDD as a way to take the businesses use cases and turn then directly into tests which the developer can then turn into either unit test or integration tests.
BDD seems to me to be not just a way to help drive the developers understanding of what the business wants, but also a way to ensure as much a spossible that those requirements are accurately represented.
My view that if you are dealing with mocks then you are doing unit tests. You need both unit testing to test out the details of a classes operation, and integrations to test out that the class plays well with others. I find developers often get infused between the two, but it's best to be as clear as possible and keep there separate from each other.
I'd like to brush my brain to avoid confusions. In few words, what can be said about Mocking process in TDD
What's the GREAT idea behind MOCKING?
Mocking frameworks are meant to be used only to avoid accessing DB during tests or they can be used for something else?
For new comers (like me), are all the frameworks equal or I need to choose one for this or that reason?
In addition to eliminating databases and other slow or ancillary concerns from the unit being tested, mocking allows you to start writing tests for a class without having to implement any collaborating classes.
As you design some piece of functionality, you'll realize that you need some other class or service, in order to stick to the single responsibility principle, but then you'll have to implement those to get the first one working, which in turn will demonstrate the need for still more classes.
If you can mock or stub those dependencies, then you can create the interfaces upon which that first class will rely, without actually having to implement anything outside of that class -- just return canned results from stubs of the interfaces.
This is an essential component to a test-first approach.
The GREAT idea: LIMIT THE SCOPE OF YOUR TESTS. By removing dependencies you remove the risk of test failures because of dependencies. That way you can focus on the correctness of the code that USES those dependencies.
Mocking DB's is very common but you can mock any dependency with an interface. In a recent project we mocked a web service, for example. You might even want to mock another business object just to make sure that you aren't relying on the correctness of the logic in that object.
I'd choose whichever one seems easiest to use. Moq is really nice.
I suggest you start here:
Mocks are not Stubs
It probably is the article that got me thinking the right way about Mocks. Sure the mocked object is usually heavy (otherwise it may not be worth mocking) but it doesn't have to be heavy in the sense that has some strong reliance on an external system like a database. It can be just a complex piece that you need to isolate to effectively be testing only your class and not the dependency.
Recently there has been quite some hype around all the different mocking frameworks in the .NET world. I still haven't quite grasped what is so great about them. It doesn't seem to be to hard to write the mocking objects I need myself. Especially with the help of Visual Studio I quickly can write a class that implements the interface I want to mock (it auto-generates almost everything for me) and then write an implementation for the method(s) I need for my test. Done! Why going through the hassle of understanding a mocking framework for the sole purpose of saving a few lines of code. Or is a mocking framework not only about saving lines of code?
Once I finally got the hang of mock objects, I realized that they're essential for unit testing for the same reason that double blind testing or control groups are essential for scientific trials: they isolate what you're actually testing.
If you're testing a class which has quite a bit of interaction via other interfaces, you not only save the lines of code on having to mock each and every interface, but you also gain the ability to do things like "throw an exception if an unexpected method is called" or "exception if these methods are called out of order". You can get remarkably sophisticated with mock frameworks, and though I'll quickly admit there's a large learning curve, when you get up to speed they'll help make your unit tests more thorough without being bloated.
You actually identified one of the key points of a mock framework in your question. The fact that you code the mocks yourself is not something the developer should be concerned with. The mocking frameworks give you implementations of interfaces programatically, plus they are functional (based on your setup of the mock).
What do you do if you are testing an ICustomerDAO, for example, and you want to test some method 14 times each with different outcomes? Implement 14 different classes manually? I doubt that anyone would want to do that.
Mocks give you the power to define what will happen with parts of your classes when you are not concerned with whether or not they will actually work, like throwing exceptions whenever you want them to, returning zero results and making sure you handle that correctly, etc...
They are a great unit testing tool.
Previous questions that may help:
What is a mock and when should you use it?
Mockist vs classical TDD
I find that using a mocking framework allows me to generate tests a lot faster and with better verification that what I expect to happen in the test actually is happengin. I have in the past implemented stubs or fakes myself. I found that I needed to generate stubs specific to the test that I wanted and this took a lot of time. I can create the same test much faster using a mocking framework. The good ones support the generation of fakes, stubs or mocks with straightforward syntax.
It takes a while to get the hang of it, I avoided it for a while but now wouldn't try to work without a mocking framework for the reasons #Chamelaeon states.
Roy Osherove had a poll about Mock Frameworks and down in the comment section, there is a discussion (albeit brief) about whether one needs a Mock Framework or not.
I personally have been manually doing exactly as you stated and it has worked well enough, but this has mainly been out of habit rather than a closely-held opinion on mock frameworks in general.
Well I certainly don't think that you NEED a mocking framework. It's a framework like any other, and it's ultimately designed to save you some time and effort. You can also do things like roll your own common data structures like stacks and queues, but isn't it generally easier to just use the ones built into the class libraries that ship with the compiler/IDE of your language of choice?
I'm sure there are other compelling reasons for using mocking frameworks, though I'd leave it to the TDD and unit testing gurus to answer.
For the same reason you wouldn't try to write unit tests without NUnit. A mocking framework will assist you in verifying state and behavior over hundreds of unit tests. It's worth the 2 weeks or so of pain to get up to speed and really helps you focus on what needs to be tested.
One thing that troubles me about a mocking framework is that "what a function should o/p given an i/p" via
when(mock.someMethod("some arg")).thenReturn("something");
statement is spread across many unit test classes.
Let me elaborate with an example. Lets say there was a DAO Interface function getEmp(int EmpID) which was returning an Employee Object when passed an Employee ID as a parameter. Assume that this function was being mocked by 10 different unit test classes. Now if in the future, this function were changed to return a newer version of the Employee Object, one would have to go to each of the 10 different classes to update this change.
The disadvantages are as follows...
a) I don't know how to figure out all the classes which mock this function so that I can go update this change.
b) My existing test cases which consumes the mock DAO object continue to be blissfully unaware of the changes that have happened to the DAO Interface because the mock has not changed and hence continue to be green.
Ideally, if I were to have coded a single mock class myself and consumed it everywhere, I would have just one place to update for the newer version of the Employee object. Also, once I update at this one place, all my existing test cases which consume the mock would break and I would then know exactly what places I need to go and do an update for the new Employee Object.
Any thoughts on my views..
One of the good things about a mocking framework is that it allows setting expectations on the objects being mocked. With the expectations I can then set up all sorts of conditions to exercise the code thats being tested.
An isolation framework or mocking framework allows you to test the code you want, without its dependencies. It makes for short running tests, allows you to debug quickly, and easily build a safety net of tests around the code. Different frameworks have different features, and as said before - it's a tool, and you should select the right tool for the job.
I've use rhino mocks for a mocking framework. I and 5 other developers used it on a large enterprise application that was an 8 month project. We used tdd on the project. Was it worth it? I guess. Was there such a massive huge selling point to using mocks that I have to use it on every project? In my opinion, no. It is not something that is necessary, it is just a tool that you can use if you want to try it out. Some projects you can roll out your own mock classes as some here say they prefer - it is easier. Other projects are larger and may require a mocking framework. The key word (in my opinion) is MAY require... how much code coverage do you require? To me, that is another consideration to using mocks. The project I did with tdd/rhino mocks we were required to have 80% code coverage so the mocks helped us attain that. If our code coverage requirements were less, for example 40%, we probably would have not used a mocking framework and just wrote our own mock classes as others mention they do.
Which scenarios, areas of an application/system, etc. are best suited for 'classic' state based testing versus using mock objects?
I'm going to tackle it from a TDD/BDD perspective, where the tests are driving the designs.
First off it depends what style of design you buy into, because remember this is about design first. As Martin Fowler discusses in this excellent article Mocks Aren't Stubs there are two school of thoughts and they do produce different types of designs.
If you want to buy into the mockist approach then I highly suggest you start by looking at the mockobjects site and their article Mock Roles, Not Objects. They also have a book coming out.
The truth is even if you do believe that the mock-first style of design is not for you, or that you don't want to do it right through your application (e.g. not when testing your domain/service layer) then you will still want to use test doubles. xUnit Test Patterns explains the different types of test doubles and their purposes.
Personally I never mock domain classes, so never mocking entities/value objects. However I've been trying out the mock objects style approach a lot recently, it does change my designs a bit and I find the working style comfortable. Working in this style in an MVC app I'll probably start with an automated acceptance test, then I'll write a controller test that mocks out any non-domain objects (e.g. repositories/services), then I'll move down to testing those repositories/services again mocking out their dependencies. I stop when I reach a class with no troublesome dependencies such as a domain entities/value object. I could go on and test against specific role interfaces which are then implemented by my domain classes, which is what the mockobjects guys would recommend, but I don't currently see a lot of value in that approach.
Obviously it is worth adding that design for test is important here, but remember that although 90% of IoC/mocking/DIP examples show interface-implementation pairs (ICustomerRepository/CustomerRepository) there is a lot of value in instead looking for role interfaces.
You should be using mocks for dependencies. I don't think that its an either-or; Usually you will create mocks for dependencies, set expectations (whether it is calls or state) on them, then run the unit under test. Then you would check its state, and verify the expectations on the mocks, afterwards.
Using mock objects doesnt mean you're not doing state based testing.
When using services, whether my own or third party, I design to interfaces as a matter of course, so I tend to focus on the interactions there. This encourages me to design minimal interfaces.
I check state on anything focusing on value objects, just as simple calculations, lookups, and the like.
The next time you find yourself designing something that typically follows Model/View/Controller-or-Presenter, I highly recommend trying the Presenter First approach (Google it) using interfaces for the Model and View. This will give you a great feel for how to use stubs/mocks effectively.
Its a matter of style.. Mockist vs Classic TDDers.
Personally.. I'd go testing real classes as far as possible. Tone down on Mocks as much as possible; only to decouple things like IO (filesystems, DB Connections, network), Third party components, etc. things that are slow/difficult to get under test.
As an experienced TDD'er, this is a question that I'm frequently asked by other developers. For me, it's a mistake to get sucked into a Mockist vs. Classicist debate, as such a discussion is misleading. State-based and behaviuour-based unit testing are two different tools in your toolbox, and there's no reason why they should be mutually exclusive.
State-based unit testing is suitable when you want to query the internal properties of an object after talking to its external interface. If one or more of its collaborators involve potentially expensive calls to other objects, then by all means stub those calls and simply disregard collaborators.
Behaviour-based unit testing is a good idea when you want to consider the "how" of a unit test and focus upon discoving relationships between objects, as opposed to the traditional "what" questions of a state-based unit test. If you wish to assert that collaborators are used in a certain way and/or sequential order, use mocks - stubs that provide assertions.
I suggest you focus upon standard unit testing practices - exercise as little production code as possible, and have one assertion per test. This will force you to think "what exactly is it that I want to exercise and assert in this test?", and the answer to that question will help you choose the correct unit testing tools.