How do mock frameworks work? - unit-testing

If I was to write a mocking library, how would this work (in other words, how do "they work?)?
One of the things which I wonder is that you are always setting expectations so really you need to compare the expectation to what the method does at runtime, so I assume reflection (resolving types at runtime) is required.
Also, when using the term "mock object", is the object stubbed out or would it be an object with pre-set expectations?
When I think how I would write my own implementation of a framework/technique, like mock objects, I realise how much I really know (or don't know) and what I would trip up on: If the mock object is pre-programmed to return set expectations and you don't call the actual real object, then wouldn't the result always be the same? Eg:
[TestMethod, Isolated]
public void FakeReturnValueByMethodArgs()
{
var fake = Isolate.Fake.Instance<ClassToIsolate>();
// MethodReturnInt will return 10 when called with arguments 3, "abc"
Isolate.WhenCalled(()=> fake.MethodReturnInt(3, " abc")).WithExactArguments().WillReturn(10);
// MethodReturnInt will return 50 when called with arguments 3, "xyz"
Isolate.WhenCalled(()=> fake.MethodReturnInt(3, "xyz")).WithExactArguments().WillReturn(50);
Assert.AreEqual(10, fake.MethodReturnInt(3, "abc"));
Assert.AreEqual(50, fake.MethodReturnInt(3, "xyz"));
}
Wouldn't this always return true?

The idea with mocking frameworks is to mock out dependencies, and not the actual classes under test. For your example, your test will always return true, because really you're only testing the mocking framework and not your actual code!
A real world mock would look more like this:
[TestMethod, Isolated]
public void FakeReturnValueByMethodArgs() {
var fake = Isolate.Fake.Instance<DependencyClass>();
// MethodReturnInt will return 10 when called with arguments 3, "abc"
Isolate.WhenCalled(()=> fake.MethodReturnInt(3, "abc")).WithExactArguments().WillReturn(10);
var testClass = new TestClass(fake);
testClass.RunMethod();
// Verify that the setup methods were execute in RunMethod()
// Not familiar with TypeMock's actual method to do this...
IsolatorExtensions.VerifyInstanceWasCalled(fake);
// Or assert on values
Assert.AreEqual(10, testClass.AProperty);
}
Notice how the mock is passed into the TestClass and a method run on it.
You can read The Purpose of Mocking to get a better idea of how mocking works.
Update: Explanation why you're testing only the mocking framework:
What you've done is create a method MethodReturnInt with the mocking framework using Isolate.WhenCalled(). When you call MethodRecturnInt in the Assert, the code will run the delegate () => fake.MethodReturnInt() and return 10. The mocking framework is effectively creating a method (albeit dynamically) that would look something like this:
public void MethodReturnInt(int value, string value2) {
Assert.Equal(3, value);
Assert.Equal("abc", value2);
return 10;
}
It's a bit more complicated than that, but this is the general idea. Since you never run any code other than the creation of 2 methods and then asserts on those two methods, you're not testing your own code and therefore only testing the mocking framework.

Yes, it will always return true. Mock objects should be used when the class under test requires another class implementation that you don't want to involve in the test run. This is most useful when it's a class that uses interfaces with multiple implementations, or there are complex/expensive/external services that you don't want to set up.
In the above code, you're mocking the class that you're "testing".
Another way of thinking about it is that the mock behaviours you record are black-box (implementation) assertions, where Assert.* are white-box (api) assertions.

You have the right idea. You will often find that they have a couple of modes of operation. If you're worried about your method not getting called or it not getting called in the right order there is quite often a 'strict' mode that causes the mock framework to throw an exception if the method isn't called by the end of the test, or is called with the wrong parameters etc.
Most of the frameworks have thought of those sorts of issues so you just need to find out how to configure it for your scenario.

One way to look at how mock system work is just look at times when you need an object but you don't want to use the real class but instead want it to give you some specific kind of data that it wouldn't (or won't do so reliably). So if you see:
Assert.IsTrue(myLogic.IsNoon(time))
you can see how the assert would want the time object to always be noon. . . well you can't do that with a real object reliably. So you need a stand-in. You can make a fake class just for the test, but that's sort of heavy. Mock frameworks are a shortcut.

Related

How to best test a service that who's effects get over-written by a stubbed out dependency

First off, my example is in php but this is not a php question, just a question on testing best practices
So I have this function that I would like to test
public function createNewTodo(CreateTodoQuery $query): TodoResponseObject
{
$new_todo = TodoFactory::createNew($query->getUserId(), $query->getTitle())
->withDescription($query->getDescription());
$new_todo = $this->todo_repository->save($new_todo);
return TodoResponseObject::fromDomain($new_todo);
}
In order to test this function, I will need to stub out my dependency (todo_repository). I want to have one test that asserts that what I get back is an instance of a TodoResponseObject. Simple enough
Now the challenging bit: I want to assert that the todo object that gets created with the parameters set in the query. Since I'm going to be stubbing out the todo_repository, I can't actually do that, as my test will just assert on the values of what I configured my stub to return. I could do something like an assertCalledWith type deal, but then at that point I'm going into the anti-pattern of testing that is "testing implementation, not functionality".
So how best could I get around this, and what would be the best way to test this?
See Sandi Metz, Magic Tricks of Testing
If you want to test that your code sent the right message to the factory, then the usual answer is to use a test double (a mock, or a spy) that tracks the messages sent to it so that you can verify them later.
This might require changing the design of your code so that you can more easily substitute one factory implementation for another (for instance, by wrapping a decorator around the "real" factory method).
Another possibility is to split the factory invocation into a separate method, and test that method's handling of the parameters
public function createTodo(CreateTodoQuery $query) {
return TodoFactory::createNew($query->getUserId(), $query->getTitle())
->withDescription($query->getDescription());
}
Changing the design of your implementation so that it better fits with your testing is normal in tdd.

Unit Testing Controllers in PHP - Should I mock everything?

Basically, i have an application written in Zend Framework 2.
My Controllers are pretty good tested, and while i made this experience i asked my self:
"Should i mock everything i can possibly mock?"
For example:
public function someAction()
{
$form = new SomeForm();
if ($this->getRequest()->isPost()) {
$form->setData($this->getRequest()->getPost());
if ($form->isValid()) {
doSomething();
}
}
}
Now: Should i mock the form? I learned its the best to mock as much as possible.
On the other hand, i've read that there is something called "overmocking" which basically means, that if you mock too much, its bad because the test might be broken.
I think you've asked two separate questions here, so I'll try to answer them both.
Should I mock everything?
In general, you should mock as little as possible to test the functionality. If you mock as much as possible, you don't ever truly see how your methods interact with each other (not integrate, as integration testing is a distinct concept).
How should I test this controller action?
In your example, and in line with my first answer, I would only "mock" the request object, and even in this case, I wouldn't actually make a mock (in the using getMock(..) sense of things), I would just construct a Request object manually. Apologies if this is not syntactically correct, I am writing this off my head:
$request = new \Zend\Http\Request();
$request->setPost(new Parameters([/* array with whatever parameters your form might need */]));
$request->setMethod('POST');
$controller = new YourController();
$controller->setRequest($request); // you may need to use a \ReflectionProperty to do this
$controller->someAction();
I would create the POST parameters using data providers to test a variety of different inputs to check that what you expect to happen will happen.
If some undesired effect would happen by calling doSomething(), then it may help if you mocked that method. Because doSomething() appears to be just a function rather than a class method, you may need some different code. However, for simplicity, lets tweak your initial example to be a call to $this->doSomething(), our test would change to something like this:
// set up $request as before
$controller = $this->getMock('\Your\Vendor\Controller\YourController', ['doSomething']);
$controller->expects($this->once())->method('doSomething');
$controller->setRequest($request); // you may need to use a \ReflectionProperty to do this
$controller->someAction();
This asserts that the "doSomething" method is called (if you want to verify that it's called).

Mocking vs. Spying in mocking frameworks

In mocking frameworks, you can mock an object or spy on it. What's the difference between the two and when would/should I use one over the other?
Looking at Mockito, for example, I see similar things being done using spies and mocks, but I am unsure as to the distinction between the two.
Mock object replace mocked class entirely, returning recorded or default values. You can create mock out of "thin air". This is what is mostly used during unit testing.
When spying, you take an existing object and "replace" only some methods. This is useful when you have a huge class and only want to mock certain methods (partial mocking). Let me quote Mockito documentation:
You can create spies of real objects. When you use the spy then the real methods are called (unless a method was stubbed).
Real spies should be used carefully and occasionally, for example when dealing with legacy code.
When in doubt, use mocks.
I'll try to explain using an example here:
// Difference between mocking, stubbing and spying
#Test
public void differenceBetweenMockingSpyingAndStubbing() {
List list = new ArrayList();
list.add("abc");
assertEquals(1, list.size());
List mockedList = spy(list);
when(mockedList.size()).thenReturn(10);
assertEquals(10, mockedList.size());
}
Here, we had initial real object list, in which we added one element and expected size to be one.
We spy real object meaning that we can instruct which method to be stubbed. So we declared that we stubbed method - size() on spy object which will return 10, no matter what is actual size.
In a nutshell, you will spy real object and stub some of the methods.
Mockito warns that partial mocking isn't a good practice and you should revise your Object Oriented architecture. Spy (or partial mocking) is recommended to test legacy code.
Based on Mocks Aren't Stubs by Martin Fowler:
Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test.
Spies are stubs that also record some information based on how they were called. One form of this might be an email service that records how many messages it was sent.
Mocks are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive.
In Mockito if you assign any object to instance variable of Mock Object then does not affect on Mock Object.
But in case of Spy, if you assign any object to instance variable of Spy Object then does affect on Spy Object because of Spy act like real-time object modification.
For a reference example are
#RunWith(MockitoJUnitRunner.class)
public class MockSpyExampleTest {
#Mock
private List<String> mockList;
#Spy
private List<String> spyList = new ArrayList();
#Test
public void testMockList() {
//by default, calling the methods of mock object will do nothing
mockList.add("test");
assertNull(mockList.get(0));
}
#Test
public void testSpyList() {
//spy object will call the real method when not stub
spyList.add("test");
assertEquals("test", spyList.get(0));
}
}
Reference: http://javapointers.com/tutorial/difference-between-spy-and-mock-in-mockito/
When using mock objects, the default behavior of the method when not stub is do nothing. Simple means, if its a void method, then it will do nothing when you call the method or if its a method with a return then it may return null, empty or the default value.
While in spy objects, of course, since it is a real method, when you are not stubbing the method, then it will call the real method behavior. If you want to change and mock the method, then you need to stub it.
Spies have two definitions. One, is where the real method is called, another where, no functionality is called and only null or null equivalent values are returned, but methods were called, and they're state was recorded, commonly like, method x was called y times.
If we want to avoid calling external services and just want to test the logic inside the method then use mock.
If we want to call the external services and use the real dependency i.e to run the program as it is and just stub specific methods then use spy

What is the unit testing strategy for method call forwarding?

I have the following scenario:
public class CarManager
{
..
public long AddCar(Car car)
{
try
{
string username = _authorizationManager.GetUsername();
...
long id = _carAccessor.AddCar(username, car.Id, car.Name, ....);
if(id == 0)
{
throw new Exception("Car was not added");
}
return id;
} catch (Exception ex) {
throw new AddCarException(ex);
}
}
public List AddCars(List cars)
{
List ids = new List();
foreach(Car car in cars)
{
ids.Add(AddCar(car));
}
return ids;
}
}
I am mocking out _reportAccessor, _authorizationManager etc.
Now I want to unittest the CarManager class.
Should I have multiple tests for AddCar() such as
AddCarTest()
AddCarTestAuthorizationManagerException()
AddCarTestCarAccessorNoId()
AddCarTestCarAccessorException()
For AddCars() should I repeat all previous tests as AddCars() calls AddCar() - it seems like repeating oneself? Should I perhaps not be calling AddCar() from AddCars()? < p/>
Please help.
There are two issues here:
Unit tests should do more than test methods one at a time. They should be designed to prove that your class can do the job it was designed for when integrated with the rest of the system. So you should mock out the dependencies and then write a test for each way in which you class will actually be used. For each (non-trivial) class you write there will be scenarios that involve the client code calling methods in a particular pattern.
There is nothing wrong with AddCars calling AddCar. You should repeat tests for error handling but only when it serves a purpose. One of the unofficial rules of unit testing is 'test to the point of boredom' or (as I like to think of it) 'test till the fear goes away'. Otherwise you would be writing tests forever. So if you are confident a test will add no value them don't write it. You may be wrong of course, in which case you can come back later and add it in. You don't have to produce a perfect test first time round, just a firm basis on which you can build as you better understand what your class needs to do.
Unit Test should focus only to its corresponding class under testing. All attributes of class that are not of same type should be mocked.
Suppose you have a class (CarRegistry) that uses some kind of data access object (for example CarPlatesDAO) which loads/stores car plate numbers from Relational database.
When you are testing the CarRegistry you should not care about if CarPlateDAO performs correctly; Since our DAO has it's own unit test.
You just create mock that behaves like DAO and returns correct or wrong values according to expected behavior. You plug this mock DAO to your CarRegistry and test only the target class without caring if all aggregated classes are "green".
Mocking allows separation of testable classes and better focus on specific functionality.
When unittesting the AddCar class, create tests that will exercise every codepath. If _authorizationManager.GetUsername() can throw an exception, create a test where your mock for this object will throw. BTW: don't throw or catch instances of Exception, but derive a meaningful Exception class.
For the AddCars method, you definitely should call AddCar. But you might consider making AddCar virtual and override it just to test that it's called with all cars in the list.
Sometimes you'll have to change the class design for testability.
Should I have multiple tests for
AddCar() such as
AddCarTest()
AddCarTestAuthorizationManagerException()
AddCarTestCarAccessorNoId()
AddCarTestCarAccessorException()
Absolutely! This tells you valuable information
For AddCars() should I repeat all previous tests as AddCars() calls AddCar() - it seems
like repeating oneself? Should I perhaps not be calling AddCar() from AddCars()?
Calling AddCar from AddCars is a great idea, it avoids violating the DRY principle. Similarly, you should be repeating tests. Think of it this way - you already wrote tests for AddCar, so when testing AddCards you can assume AddCar does what it says on the tin.
Let's put it this way - imagine AddCar was in a different class. You would have no knowledge of an authorisation manager. Test AddCars without the knowledge of what AddCar has to do.
For AddCars, you need to test all normal boundary conditions (does an empty list work, etc.) You probably don't need to test the situation where AddCar throws an exception, as you're not attempting to catch it in AddCars.
Writing tests that explore every possible scenario within a method is good practice. That's how I unit test in my projects. Tests like AddCarTestAuthorizationManagerException(), AddCarTestCarAccessorNoId(), or AddCarTestCarAccessorException() get you thinking about all the different ways your code can fail which has led to me find new kinds of failures for a method I might have otherwise missed as well as improve the overall design of the class.
In a situation like AddCars() calling AddCar() I would mock the AddCar() method and count the number of times it's called by AddCars(). The mocking library I use allows me to create a mock of CarManager and mock only the AddCar() method but not AddCars(). Then your unit test can set how many times it expects AddCar() to be called which you would know from the size of the list of cars passed in.

When should I mock?

I have a basic understanding of mock and fake objects, but I'm not sure I have a feeling about when/where to use mocking - especially as it would apply to this scenario here.
Mock objects are useful when you want to test interactions between a class under test and a particular interface.
For example, we want to test that method sendInvitations(MailServer mailServer) calls MailServer.createMessage() exactly once, and also calls MailServer.sendMessage(m) exactly once, and no other methods are called on the MailServer interface. This is when we can use mock objects.
With mock objects, instead of passing a real MailServerImpl, or a test TestMailServer, we can pass a mock implementation of the MailServer interface. Before we pass a mock MailServer, we "train" it, so that it knows what method calls to expect and what return values to return. At the end, the mock object asserts, that all expected methods were called as expected.
This sounds good in theory, but there are also some downsides.
Mock shortcomings
If you have a mock framework in place, you are tempted to use mock object every time you need to pass an interface to the class under the test. This way you end up testing interactions even when it is not necessary. Unfortunately, unwanted (accidental) testing of interactions is bad, because then you're testing that a particular requirement is implemented in a particular way, instead of that the implementation produced the required result.
Here's an example in pseudocode. Let's suppose we've created a MySorter class and we want to test it:
// the correct way of testing
testSort() {
testList = [1, 7, 3, 8, 2]
MySorter.sort(testList)
assert testList equals [1, 2, 3, 7, 8]
}
// incorrect, testing implementation
testSort() {
testList = [1, 7, 3, 8, 2]
MySorter.sort(testList)
assert that compare(1, 2) was called once
assert that compare(1, 3) was not called
assert that compare(2, 3) was called once
....
}
(In this example we assume that it's not a particular sorting algorithm, such as quick sort, that we want to test; in that case, the latter test would actually be valid.)
In such an extreme example it's obvious why the latter example is wrong. When we change the implementation of MySorter, the first test does a great job of making sure we still sort correctly, which is the whole point of tests - they allow us to change the code safely. On the other hand, the latter test always breaks and it is actively harmful; it hinders refactoring.
Mocks as stubs
Mock frameworks often allow also less strict usage, where we don't have to specify exactly how many times methods should be called and what parameters are expected; they allow creating mock objects that are used as stubs.
Let's suppose we have a method sendInvitations(PdfFormatter pdfFormatter, MailServer mailServer) that we want to test. The PdfFormatter object can be used to create the invitation. Here's the test:
testInvitations() {
// train as stub
pdfFormatter = create mock of PdfFormatter
let pdfFormatter.getCanvasWidth() returns 100
let pdfFormatter.getCanvasHeight() returns 300
let pdfFormatter.addText(x, y, text) returns true
let pdfFormatter.drawLine(line) does nothing
// train as mock
mailServer = create mock of MailServer
expect mailServer.sendMail() called exactly once
// do the test
sendInvitations(pdfFormatter, mailServer)
assert that all pdfFormatter expectations are met
assert that all mailServer expectations are met
}
In this example, we don't really care about the PdfFormatter object so we just train it to quietly accept any call and return some sensible canned return values for all methods that sendInvitation() happens to call at this point. How did we come up with exactly this list of methods to train? We simply ran the test and kept adding the methods until the test passed. Notice, that we trained the stub to respond to a method without having a clue why it needs to call it, we simply added everything that the test complained about. We are happy, the test passes.
But what happens later, when we change sendInvitations(), or some other class that sendInvitations() uses, to create more fancy pdfs? Our test suddenly fails because now more methods of PdfFormatter are called and we didn't train our stub to expect them. And usually it's not only one test that fails in situations like this, it's any test that happens to use, directly or indirectly, the sendInvitations() method. We have to fix all those tests by adding more trainings. Also notice, that we can't remove methods no longer needed, because we don't know which of them are not needed. Again, it hinders refactoring.
Also, the readability of test suffered terribly, there's lots of code there that we didn't write because of we wanted to, but because we had to; it's not us who want that code there. Tests that use mock objects look very complex and are often difficult to read. The tests should help the reader understand, how the class under the test should be used, thus they should be simple and straightforward. If they are not readable, nobody is going to maintain them; in fact, it's easier to delete them than to maintain them.
How to fix that? Easily:
Try using real classes instead of mocks whenever possible. Use the real PdfFormatterImpl. If it's not possible, change the real classes to make it possible. Not being able to use a class in tests usually points to some problems with the class. Fixing the problems is a win-win situation - you fixed the class and you have a simpler test. On the other hand, not fixing it and using mocks is a no-win situation - you didn't fix the real class and you have more complex, less readable tests that hinder further refactorings.
Try creating a simple test implementation of the interface instead of mocking it in each test, and use this test class in all your tests. Create TestPdfFormatter that does nothing. That way you can change it once for all tests and your tests are not cluttered with lengthy setups where you train your stubs.
All in all, mock objects have their use, but when not used carefully, they often encourage bad practices, testing implementation details, hinder refactoring and produce difficult to read and difficult to maintain tests.
For some more details on shortcomings of mocks see also Mock Objects: Shortcomings and Use Cases.
A unit test should test a single codepath through a single method. When the execution of a method passes outside of that method, into another object, and back again, you have a dependency.
When you test that code path with the actual dependency, you are not unit testing; you are integration testing. While that's good and necessary, it isn't unit testing.
If your dependency is buggy, your test may be affected in such a way to return a false positive. For instance, you may pass the dependency an unexpected null, and the dependency may not throw on null as it is documented to do. Your test does not encounter a null argument exception as it should have, and the test passes.
Also, you may find its hard, if not impossible, to reliably get the dependent object to return exactly what you want during a test. That also includes throwing expected exceptions within tests.
A mock replaces that dependency. You set expectations on calls to the dependent object, set the exact return values it should give you to perform the test you want, and/or what exceptions to throw so that you can test your exception handling code. In this way you can test the unit in question easily.
TL;DR: Mock every dependency your unit test touches.
Rule of thumb:
If the function you are testing needs a complicated object as a parameter, and it would be a pain to simply instantiate this object (if, for example it tries to establish a TCP connection), use a mock.
You should mock an object when you have a dependency in a unit of code you are trying to test that needs to be "just so".
For example, when you are trying to test some logic in your unit of code but you need to get something from another object and what is returned from this dependency might affect what you are trying to test - mock that object.
A great podcast on the topic can be found here