I have just started to read Professional Test Driven Development with C#: Developing Real World Applications with TDD
I have a hard time understanding stubs, fakes and mocks. From what I understand so far, they are fake objects used for the purpose of unit testing your projects, and that a mock is a stub with conditional logic into it.
Another thing I think I have picked up is that mocks are somehow related with dependency injection, a concept which I only managed to understand yesterday.
What I do not get is why I would actually use them. I cannot seem to find any concrete examples online that explains them properly.
Can anyone please explain to me this concepts?
As I've read in the past, here's what I believe each term stands for
Stub
Here you are stubbing the result of a method to a known value, just to let the code run without issues. For example, let's say you had the following:
public int CalculateDiskSize(string networkShareName)
{
// This method does things on a network drive.
}
You don't care what the return value of this method is, it's not relevant. Plus it could cause an exception when executed if the network drive is not available. So you stub the result in order to avoid potential execution issues with the method.
So you end up doing something like:
sut.WhenCalled(() => sut.CalculateDiskSize()).Returns(10);
Fake
With a fake you are returning fake data, or creating a fake instance of an object. A classic example are repository classes. Take this method:
public int CalculateTotalSalary(IList<Employee> employees) { }
Normally the above method would be passed a collection of employees that were read from a database. However in your unit tests you don't want to access a database. So you create a fake employees list:
IList<Employee> fakeEmployees = new List<Employee>();
You can then add items to fakeEmployees and assert the expected results, in this case the total salary.
Mocks
When using mock objects you intend to verify some behaviour, or data, on those mock objects. Example:
You want to verify that a specific method was executed during a test run, here's a generic example using Moq mocking framework:
public void Test()
{
// Arrange.
var mock = new Mock<ISomething>();
mock.Expect(m => m.MethodToCheckIfCalled()).Verifiable();
var sut = new ThingToTest();
// Act.
sut.DoSomething(mock.Object);
// Assert
mock.Verify(m => m.MethodToCheckIfCalled());
}
Hopefully the above helps clarify things a bit.
EDIT:
Roy Osherove is a well-known advocate of Test Driven Development, and he has some excellent information on the topic. You may find it very useful :
http://artofunittesting.com/
They are all variations of the Test Double. Here is a very good reference that explains the differences between them: http://xunitpatterns.com/Test%20Double.html
Also, from Martin Fowler's post: http://martinfowler.com/articles/mocksArentStubs.html
Meszaros uses the term Test Double as the generic term for any kind of
pretend object used in place of a real object for testing purposes.
The name comes from the notion of a Stunt Double in movies. (One of
his aims was to avoid using any name that was already widely used.)
Meszaros then defined four particular kinds of double:
Dummy objects: are passed around but never actually used. Usually they
are just used to fill parameter lists.
Fake objects actually have working implementations, but usually take some shortcut which makes
them not suitable for production (an in memory database is a good
example).
Stubs provide canned answers to calls made during the test,
usually not responding at all to anything outside what's programmed in
for the test. Stubs may also record information about calls, such as
an email gateway stub that remembers the messages it 'sent', or maybe
only how many messages it 'sent'.
Mocks are what we are talking about here: objects pre-programmed with expectations which form a
specification of the calls they are expected to receive.
Of these kinds of doubles, only mocks insist upon behavior verification. The
other doubles can, and usually do, use state verification. Mocks
actually do behave like other doubles during the exercise phase, as
they need to make the SUT believe it's talking with its real
collaborators.
This PHP Unit's manual helped me a lot as introduction:
"Sometimes it is just plain hard to test the system under test (SUT) because it depends on other components that cannot be used in the test environment. This could be because they aren't available, they will not return the results needed for the test or because executing them would have undesirable side effects. In other cases, our test strategy requires us to have more control or visibility of the internal behavior of the SUT." More: https://phpunit.de/manual/current/en/test-doubles.html
And i find better "introductions" when looking for "test doubles" as mocks, fakes, stubs and the others are known.
Related
I'm sorry, it's a very long post. I've read almost everything on the subject, and I'm not yet convinced that it's a bad idea to partially mock the SUT to just DRY up the tests. So, I need to first address all the reasonings against it in order to avoid repetitive answers. Please bear with me.
Have you ever felt the urge to partially mock out the SUT itself, in order to make tests more DRY? Less mocks, less madness, more readable tests?!
Let's have an example at hand to discuss the subject more clearly:
class Sut
{
public function fn0(...)
{
// Interacts with: Dp0
}
public function fn1(...)
{
// Calls: fn0
// Interacts with: Dp1
}
public function fn2(...)
{
// Calls: fn0
// Interacts with: Dp1
}
public function fn3(...)
{
// Calls: fn2
}
public function fn4(...)
{
// Calls: fn1(), fn2(), fn3()
// Interacts with: Dp2
}
}
Now, let's test the behavior of the SUT. Each fn*() is representing a behavior of the class under test. Here, I'm not trying to unit-test each single method of the SUT, but its exposed behaviors.
class SutTest extends \PHPUnit_Framework_Testcase
{
/**
* #covers Sut::fn0
*/
public function testFn0()
{
// Mock Dp0
}
/**
* #covers Sut::fn1
*/
public function testFn1()
{
// Mock Dp1, which is a direct dependency
// Mock Dp0, which is an indirect dependency
}
/**
* #covers Sut::fn2
*/
public function testFn2()
{
// Mock Dp1 with different expectations than testFn1()
// Mock Dp0 with different expectations
}
/**
* #covers Sut::fn3
*/
public function testFn3()
{
// Mock Dp1, again with different expectations
// Mock Dp0, with different expectations
}
/**
* #covers Sut::fn4
*/
public function testFn4()
{
// Mock Dp2 which is a direct dependency
// Mock Dp0, Dp1 as indirect dependencies
}
}
You get the terrible idea! You need to keep repeating yourself. It's not DRY at all. And as the expectations of each mock object may differ for each test, you can't just mock out all the dependencies and set expectations once for the whole testcase. You need to be explicit about the behavior of the mock for each test.
Let's also have some real code to see what it would look like when a test needs to mock out all dependencies through the code path it's testing:
/** #test */
public function dispatchesActionsOnACollectionOfElementsFoundByALocator()
{
$elementMock = $this->mock(RemoteWebElement::class)
->shouldReceive('sendKeys')
->times(3)
->with($text = 'some text...')
->andReturn(Mockery::self())
->mock();
$this->inject(RemoteWebDriver::class)
->shouldReceive('findElements')
->with(WebDriverBy::class)
->andReturn([$elementMock, $elementMock, $elementMock])
->shouldReceive('getCurrentURL')
->zeroOrMoreTimes();
$this->inject(WebDriverBy::class)
->shouldReceive('xpath')
->once()
->with($locator = 'someLocatorToMatchMultipleElements')
->andReturn(Mockery::self());
$this->inject(Locator::class)
->shouldReceive('isLocator', 'isXpath')
->andReturn(true, false);
$this->type($text, $locator);
}
Madness! In order to test a small method, you find yourself writing such a test that is extremely non-readable and coupled to the implementation details of 3 or 4 other dependant methods in its way up the chain. It's even more terrible when you see the whole testcase; lot's of those mock blocks, duplicated to set different expectations, covering different paths of code. The test is mirroring the implementation details of few others. It's painful.
Workaround
OK, back to the first pseudo-code; While testing fn3(), you start thinking, what if I could mock the call to fn2() and put a stop to all the mocking madness? I make a partial mock of the SUT, set expectations for the fn2() and I make sure that the method under test interacts with fn2(), correctly.
In other words, to avoid excessive mocking of external dependencies, I focus on just one behavior of the SUT (might be one or couple of methods) and make sure it behaves correctly. I mockout all other methods that belong to other behaviors of the SUT. No worries about them, they all have their own tests.
Contrary reasonings
One might discuss that:
The problem with stubbing/mocking methods in a class is that you are violating the encapsulation. Your test should be checking to see whether or not the external behaviour of the object matches the specifications. Whatever happens inside the object is none of its business. By mocking the public methods in order to test the object, you are making an assumption about how that object is implemented.
When unit-testing, it's rare that you always deal with such behaviors that are fully testable by providing input and expecting output; seeing them as black boxes. Most of the time you need to test how they interact with one another. So, we need to have, at least, some information about the internal implementation of the SUT, to be able to fully test it.
When we mock a dependency, we're ALREADY making an assumption about how the SUT works. We're binding the test to the implementation details of the SUT. So, now that we're deep in mud, why not mocking an internal method to make our lives easier?!
Some may say:
Mocking the methods is taking the object (SUT) and breaking it into two pieces. One piece is being mocked while the other piece is being tested. What you are doing is essentially an ad-hoc breaking up of the object. If that's the case, just break up the object already.
Unit tests should treat the classes they test as black boxes. The only thing which matters is that its public methods behave the way it is expected. How the class achieves this through internal state and private methods does not matter. When you feel that it is impossible to create meaningful tests this way, it is a sign that your classes are too powerful and do too much. You should consider to move some of their functionality to separate classes which can be tested separately.
If there are arguments that support this separation (partially mock the SUT), the same arguments can be used to refactor the class into two classes and this is exactly what you should to then.
If it's a smell for SRP, yeah right, the functionality can be extracted into another class and then you can easily mock that class and go happily home. But it's not the case. The SUT's design is ok, it has no SRP issues, it's small, it does one job and ad-heres to SOLID principles. When looking at SUT's code, there's no reason you want to break the functionality into some other classes. It's already broken into very fine pieces.
How come when you look at SUT tests, you decide to break the class? How come it's ok to mock out aaaaall those dependencies along the way, when testing fn3() but it's not ok to mock the only real dependency it has (even though it's an internal one)? fn2(). Either way, we're bound to the implementation details of the SUT. Either way, the tests are fragile.
It's important to notice why we want to mock those methods. We just want easier testing, less mocks, while maintaining the absolute isolation of the SUT (more on this later).
Some other might reason:
The way I see it, an object has external and internal behaviors. External behavior includes returns values, calls into other objects, etc. Obviously, anything in that category should be tested. But internal behavior shouldn't really be tested. I don't write tests directly on the internal behavior, only indirectly through the external behavior.
Right, I do that, too. But we're not testing the internals of SUT, we're just exercising its public API and we want to avoid excessive mocking.
Reasoning states that, External behavior includes calls into other objects; I agree. We're tryinng to test the SUT's external calls here as well, just by early-mocking the internal method that makes the interaction. That mocked method(s) has the its tests already.
One other reasons:
Too many mocks and already perfectly broken into multiple classes? You are over-unit-testing by unit-testing what should be integration tested.
The example code has just 3 external dependencies and I don't think that it's too much. Again, it's important to notice why I want to partially mock the SUT; only and only for easier testing, avoiding excessive mocks.
By the way, the reasoning might be true somehow. I might need to do integration testing in some cases. More on this in the next section.
The last one says:
These are all tests man, not production code, they don't need to be DRY!
I've actually read something like this! And I simply don't think so. I need to put my lifetime into use! You, too!
Bottom-line: To mock, or not to mock?
When we choose to mock, we're writing whitebox unit-tests. We're bounding tests to the implementation details of the SUT, more or less. Then, if we decide to go down the PURE way, and radically maintain the isolation of the SUT, soon or later, we find ourselves in the hell of mocking madness... and fragile tests. Ten months into the maintenance, you found yourself serving unit-tests instead of them serving you! You find yourself re-implementing multiple tests for a single change in the implementation of one SUT method. Painful right?
So, if we're going this way, why not partially mock the SUT? Why not making our lives waaaaaaaaay easier? I see no reason not to do so? Do you?
I've read and read, and finally came across this article by Uncle Bob:
https://8thlight.com/blog/uncle-bob/2014/05/10/WhenToMock.html
To qoute the most important part:
Mock across architecturally significant boundaries, but not within those boundaries.
I think it's the remedy to all the mocking madness I told you about. There's no need to radically maintain the isolation of the SUT as I've learnt blindly. Even though it may work for most of the time, it also may force you to live in your private mocking hell, banging your head against the wall.
This little gem piece of advice, is the only reasoning that makes sense to not to partially mock the SUT. In fact, it's the exact opposite of doing so. But now the question would be, isn't that integration testing? Is that still called unit-testing? What's the UNIT here? Architecturally significant boundaries?
Here's another article by Google Testing team, implicitly suggesting the same practice:
https://testing.googleblog.com/2013/05/testing-on-toilet-dont-overuse-mocks.html
To recap
If we're going down the pure isolation way, assuming that the SUT is already broken into fine pieces, with minimum possible external deps, is there any reason not to partially mock the SUT? In order to avoid the excessive mocking and to make unit-tests more DRY?
If we take the advice of Uncle Bob to the heart and only "Mock across architecturally significant boundaries, but not within those boundaries.", is that still considered unit testing? What's the unit here?
Thank you for reading.
P.S. Those contrary reasonings are more or less from existing SO answers or articles I found on the subject. Unfortunately, I don't currently have the refs to link to.
Unit tests don't have to be isolated unit tests, at least if you accept the definition promoted by authors like Martin Fowler and Kent Beck. Kent is the creator of JUnit, and probably the main proponent of TDD. These guys don't do mocking.
In my own experience (as long-time developer of an advanced Java mocking library), I see programmers misusing and abusing mocking APIs all the time. In particular, when they think that partially mocking the SUT is a valid idea. It is not. Making test code more DRY shouldn't be an excuse for over-mocking.
Personally, I favor integration tests with minimal or (preferably) no mocking. As long as your tests are stable and run fast enough, they are ok. The important thing is that tests don't become a pain to write, maintain, and more importantly don't discourage the programmers from running them. (This is why I avoid functional UI-driven tests - they tend to be a pain to run.)
I have started using moq for mocking. Can someone explain me the concept of strict and non-strict mocks? How can they can be used in moq?
edit:
in which scenario do we use which type of mock?
I'm not sure about moq specifically, but here's how strict mocks work in Rhino. I declare that I expect a call to foo.Bar on my object foo:
foo.Expect(f => f.Bar()).Returns(5);
If the calling code does
foo.Bar();
then I'm fine because the expectations are exactly met.
However, if the calling code is:
foo.Quux(12);
foo.Bar();
then my expectation failed because I did not explicitly expect a call to foo.Quux.
To summarize, a strict mock will fail immediately if anything differs from the expectations. On the other hand, a non-strict mock (or a stub) will gladly "ignore" the call to foo.Quux and it should return a default(T) for the return type T of foo.Quux.
The creator of Rhino recommends that you avoid strict mocks (and prefer stubs) because you generally don't want your test to fail when receiving an unexpected call as above. It makes refactoring your code much more difficult when you have to fix dozens of test that relied on the exact original behavior.
Ever come across Given / When / Then?
Given a context
When I perform some events
Then an outcome should occur
This pattern appears in BDD's scenarios, and is also relevant for unit tests.
If you're setting up context, you're going to use the information which that context provides. For instance, if you're looking up something by Id, that's context. If it doesn't exist, the test won't run. In this case, you want to use a NiceMock or a Stub or whatever - Moq's default way of running.
If you want to verify an outcome, you can use Moq's verify. In this case, you want to record the relevant interactions. Fortunately, this is also Moq's default way of running. It won't complain if something happens that you weren't interested in for that test.
StrictMock is there for when you want no unexpected interactions to occur. It's how old-style mocking frameworks used to run. If you're doing BDD-style examples, you probably won't want this. It has a tendency to make tests a bit brittle and harder to read than if you separate the aspects of behaviour you're interested in. You have to set up expectations for both the context and the outcome, for all outcomes which will occur, regardless of whether they're of interest or not.
For instance, if you're testing a controller and mocking out both your validator and your repository, and you want to verify that you've saved your object, with a strict mock you also have to verify that you've validated the object first. I prefer to see those two aspects of behaviour in separate examples, because it makes it easier for me to understand the value and behaviour of the controller.
In the last four years I haven't found a single example which required the use of a strict mock - either it was an outcome I wanted to verify (even if I verify the number of times it's called) or a context for which I can tell if I respond correctly to the information provided. So in answer to your question:
non-strict mock: usually
strict mock: preferably never
NB: I am strongly biased towards BDD, so hard-core TDDers may disagree with me, and it will be right for the way that they are working.
Here's a good article.
I usually end up having something like this
public class TestThis {
private final Collaborator1 collaborator1;
private final Collaborator2 collaborator2;
private final Collaborator2 collaborator3;
TestThis(Collaborator1 collaborator1, Collaborator2 collaborator2, Collaborator3 collaborator3) {
this.collaborator1 = collaborator1;
this.collaborator2 = collaborator2;
this.collaborator3 = collaborator3;
}
public Login login(String username) {
User user = collaborator1.getUser(username);
collaborator2.notify(user);
return collaborator3.login(user);
}
}
...and I use Strict mocks for the 3 collaborators to test login(username). I don't see how Strict Mocks should never be used.
I have a simple convention:
Use strict mocks when the system under test (SUT) is delegating the call to the underlying mocked layer without really modifying or applying any business logic to the arguments passed to itself.
Use loose mocks when the SUT applies business logic to the arguments passed to itself and passes on some derived/modified values to the mocked layer.
For eg:
Lets say we have database provider StudentDAL which has two methods:
Data access interface looks something like below:
public Student GetStudentById(int id);
public IList<Student> GetStudents(int ageFilter, int classId);
The implementation which consumes this DAL looks like below:
public Student FindStudent(int id)
{
//StudentDAL dependency injected
return StudentDAL.GetStudentById(id);
//Use strict mock to test this
}
public IList<Student> GetStudentsForClass(StudentListRequest studentListRequest)
{
//StudentDAL dependency injected
//age filter is derived from the request and then passed on to the underlying layer
int ageFilter = DateTime.Now.Year - studentListRequest.DateOfBirthFilter.Year;
return StudentDAL.GetStudents(ageFilter , studentListRequest.ClassId)
//Use loose mock and use verify api of MOQ to make sure that the age filter is correctly passed on.
}
I think understand the definition of State / Interaction based testing (read the Fowler thing, etc). I found that I started state based but have been doing more interaction based and I'm getting a bit confused on how to test certain things.
I have a controller in MVC and an action calls a service to deny a package:
public ActionResult Deny(int id)
{
service.DenyPackage(id);
return RedirectToAction("List");
}
This seems clear to me. Provide a mock service, verify it was called correctly, done.
Now, I have an action for a view that lets the user associate a certificate with a package:
public ActionResult Upload(int id)
{
var package = packageRepository.GetPackage(id);
var certificates = certificateRepository.GetAllCertificates();
var view = new PackageUploadViewModel(package, certificates);
return View(view);
}
This one I'm a bit stumped on. I'm doing Spec style tests (possibly incorrectly) so to test this method I have a class and then two tests: verify the package repository was called, verify the certificate repository was called. I actually want a third to test to verify that the constructor was called but have no idea how to do that! I'm get the impression this is completely wrong.
So for state based testing I would pass in the id and then test the ActionResult's view. Okay, that makes sense. But wouldn't I have a test on the PackageUploadViewModel constructor? So if I have a test on the constructor, then part of me would just want to verify that I call the constructor and that the action return matches what the constructor returns.
Now, another option I can think of is I have a PackageUploadViewModelBuilder (or something equally dumbly named) that has dependency on the two repositories and then I just pass the id to a CreateViewModel method or something. I could then mock this object, verify everything, and be happy. But ... well ... it seems extravagant. I'm making something simple ... not simple. Plus, controller.action(id) returning builder.create(id) seems like adding a layer for no reason (the controller is responsible for building view models.. right?)
I dunno... I'm thinking more state based testing is necessary, but I'm afraid if I start testing return values then if Method A can get called in 8 different contexts I'm going to have a test explosion with a lot of repetition. I had been using interaction based testing to pass some of those contexts to Method B so that all I have to do is verify Method A called Method B and I have Method B tested so Method A can just trust that those contexts are handled. So interaction based testing is building this hierarchy of tests but state based testing is going to flatten it out some.
I have no idea if that made any sense.
Wow, this is long ...
I think Roy Osherove recently twitted that as a rule of thumb, your tests should be 95 percent state-based and 5 percent interaction-based. I agree.
What matters most is that your API does what you want it to, and that is what you need to test. If you test the mechanics of how it achieves what it needs to do, you are very likely to end up with Overspecified Tests, which will bite you when it comes to maintainability.
In most cases, you can design your API so that state-based testing is the natural choice, because that is just so much easier.
To examine your Upload example: Does it matter that GetPackage and GetAllCertificates was called? Is that really the expected outcome of the Upload method?
I would guess not. My guess is that the purpose of the Upload method - it's very reason for existing - is to populate and serve the correct View.
So state-based testing would examine the returned ViewResult and its ViewModel and verify that it has all the correct values.
Sure, as the code stands right now, you will need to provide Test Doubles for packageRepository and certificateRepository, because otherwise exceptions will be thrown, but it doesn't look like it is important in itself that the repository methods are being called.
If you use Stubs instead of Mocks for your repositories, your tests are no longer tied to internal implementation details. If you later on decide to change the implementation of the Upload method to use cached instances of packages (or whatever), the Stub will not be called, but that's okay because it's not important anyway - what is important is that the returned View contains the expected data.
This is much more preferrable than having the test break even if all the returned data is as it should be.
Interestingly, your Deny example looks like a prime example where interaction-based testing is still warranted, because it is only by examining Indirect Outputs that you can verify that the method performed the correct action (the DenyPackage method returns void).
All this, and more, is explained very well in the excellent book xUnit Test Patterns.
The question to ask is "if this code worked, how could I tell?" That might mean testing some interactions or some state, it depends on what's important.
In your first test, the Deny changes the world outside the target class. It requires a collaboration from a service, so testing an interaction makes sense. In your second test, you're making queries on the neighbours (not changing anything outside the target class), so stubbing them makes more sense.
That's why we have a heuristic of "Stub Queries, Mock Actions" in http://www.mockobjects.com/book
I'm basically trying to teach myself how to code and I want to follow good practices. There are obvious benefits to unit testing. There is also much zealotry when it comes to unit-testing and I prefer a much more pragmatic approach to coding and life in general. As context, I'm currently writing my first "real" application which is the ubiquitous blog engine using asp.net MVC. I'm loosely following the MVC Storefront architecture with my own adjustments. As such, this is my first real foray into mocking objects. I'll put the code example at the end of the question.
I'd appreciate any insight or outside resources that I could use to increase my understanding of the fundamentals of testing and mocking. The resources I've found on the net are typically geared towards the "how" of mocking and I need more understanding of the where, why and when of mocking. If this isn't the best place to ask this question, please point me to a better place.
I'm trying to understand the value that I'm getting from the following tests. The UserService is dependent upon the IUserRepository. The value of the service layer is to separate your logic from your data storage, but in this case most of the UserService calls are just passed straight to IUserRepository. The fact that there isn't much actual logic to test could be the source of my concerns as well. I have the following concerns.
It feels like the code is just testing that the mocking framework is working.
In order to mock out the dependencies, it makes my tests have too much knowledge of the IUserRepository implementation. Is this a necessary evil?
What value am I actually gaining from these tests? Is the simplicity of the service under test causing me to doubt the value of these tests.
I'm using NUnit and Rhino.Mocks, but it should be fairly obvious what I'm trying to accomplish.
[SetUp]
public void Setup()
{
userRepo = MockRepository.GenerateMock<IUserRepository>();
userSvc = new UserService(userRepo);
theUser = new User
{
ID = null,
UserName = "http://joe.myopenid.com",
EmailAddress = "joe#joeblow.com",
DisplayName = "Joe Blow",
Website = "http://joeblow.com"
};
}
[Test]
public void UserService_can_create_a_new_user()
{
// Arrange
userRepo.Expect(repo => repo.CreateUser(theUser)).Return(true);
// Act
bool result = userSvc.CreateUser(theUser);
// Assert
userRepo.VerifyAllExpectations();
Assert.That(result, Is.True,
"UserService.CreateUser(user) failed when it should have succeeded");
}
[Test]
public void UserService_can_not_create_an_existing_user()
{
// Arrange
userRepo.Stub(repo => repo.IsExistingUser(theUser)).Return(true);
userRepo.Expect(repo => repo.CreateUser(theUser)).Return(false);
// Act
bool result = userSvc.CreateUser(theUser);
// Assert
userRepo.VerifyAllExpectations();
Assert.That(result, Is.False,
"UserService.CreateUser() allowed multiple copies of same user to be created");
}
Essentially what you are testing here is that the methods are getting called, not whether or not they actually work. Which is what mocks are supposed to do. Instead of calling the method, they just check to see if the method got called, and return whatever is in the Return() statement. So in your assertion here:
Assert.That(result, Is.False, "error message here");
This assertion will ALWAYS succeed because your expectation will ALWAYS return false, because of the Return statement:
userRepo.Expect(repo => repo.CreateUser(theUser)).Return(false);
I'm guessing this isn't that useful in this case.
Where mocking is useful is when you want to, for example, make a database call somewhere in your code, but you don't want to actually call to the database. You want to pretend that the database got called, but you want to set up some fake data for it to return, and then (here's the important part) test the logic that does something with the fake data your mock returned. In the above examples you are omitting the last step. Imagine you had a method that displayed a message to the user that said whether the new user was created:
public string displayMessage(bool userWasCreated) {
if (userWasCreated)
return "User created successfully!";
return "User already exists";
}
then your test would be
userRepo.Expect(repo => repo.CreateUser(theUser)).Return(false);
Assert.AreEqual("User already exists", displayMessage(userSvc.CreateUser(theUser)))
Now this has some value, because you are testing some actual behavior. Of course, you could also just test this directly by passing in "true" or "false." You don't even need a mock for that test. Testing expectations is fine, but I've written plenty of tests like that, and have come to the same conclusion that you are reaching - it just isn't that useful.
So in short, mocking is useful when you want to abstract away externalities, like databases, or webservice calls, etc, and inject known values at that point. But it's not often useful to test mocks directly.
You are right: the simplicity of the service makes these tests uninteresting. It is not until you get more business logic in the service, that you will gain value from the tests.
You might consider some tests like these:
CreateUser_fails_if_email_is_invalid()
CreateUser_fails_if_username_is_empty()
Another comment: it looks like a code-smell, that your methods return booleans to indicate success or failure. You might have a good reason to do it, but usually you should let the exceptions propagate out. It also makes it harder to write good tests, since you will have problems detecting whether your method failed for the "right reason", f.x. you might write the CreateUser_fails_if_email_is_invalid()-test like this:
[Test]
public void CreateUser_fails_if_email_is_invalid()
{
bool result = userSvc.CreateUser(userWithInvalidEmailAddress);
Assert.That(result, Is.False);
}
and it would probably work with your existing code. Using the TDD Red-Green-Refactor-cycle would mitigate this problem, but it would be even better to be able to actually detect that the method failed because of the invalid email, and not because of another problem.
If you write your tests before you write your code, you'll gain much more from your unit tests. One of the reasons that it feels like your tests aren't worth much is that you're not deriving the value of having your tests drive the design. Writing your tests afterwards is mostly just an exercise in seeing if you can remember everything that can go wrong. Writing your tests first causes you to think about how you would actually implement the functionality.
These tests aren't all that interesting because the functionality that is being implemented is pretty basic. The way you are going about mocking seems pretty standard -- mock the things the class under test depends on, not the class under test. Testability (or good design sense) has already led you to implement interfaces and use dependency injection to reduce coupling. You might want to think about changing the error handling, as other have suggested. It would be nice to know why, if only to improve the quality of your tests, CreateUser has failed, for instance. You could do this with exceptions or with an out parameter (which is how MembershipProvider works, if I remember correctly).
You are facing the question of "classical" vs. "mockist" approaches to testing. Or "state-verification" vs. "behaviour-verification" as described by Martin Fowler: http://martinfowler.com/articles/mocksArentStubs.html#ClassicalAndMockistTesting
Another most excellent resource is Gerard Meszaros' book "xUnit Test Patterns: Refactoring Test Code"
I know how I use these terms, but I'm wondering if there are accepted definitions for faking, mocking, and stubbing for unit tests? How do you define these for your tests? Describe situations where you might use each.
Here is how I use them:
Fake: a class that implements an interface but contains fixed data and no logic. Simply returns "good" or "bad" data depending on the implementation.
Mock: a class that implements an interface and allows the ability to dynamically set the values to return/exceptions to throw from particular methods and provides the ability to check if particular methods have been called/not called.
Stub: Like a mock class, except that it doesn't provide the ability to verify that methods have been called/not called.
Mocks and stubs can be hand generated or generated by a mocking framework. Fake classes are generated by hand. I use mocks primarily to verify interactions between my class and dependent classes. I use stubs once I have verified the interactions and am testing alternate paths through my code. I use fake classes primarily to abstract out data dependencies or when mocks/stubs are too tedious to set up each time.
You can get some information :
From Martin Fowler about Mock and Stub
Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production
Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent'.
Mocks are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive.
From xunitpattern:
Fake: We acquire or build a very lightweight implementation of the same functionality as provided by a component that the SUT depends on and instruct the SUT to use it instead of the real.
Stub : This implementation is configured to respond to calls from the SUT with the values (or exceptions) that will exercise the Untested Code (see Production Bugs on page X) within the SUT. A key indication for using a Test Stub is having Untested Code caused by the inability to control the indirect inputs of the SUT
Mock Object that implements the same interface as an object on which the SUT (System Under Test) depends. We can use a Mock Object as an observation point when we need to do Behavior Verification to avoid having an Untested Requirement (see Production Bugs on page X) caused by an inability to observe side-effects of invoking methods on the SUT.
Personally
I try to simplify by using : Mock and Stub. I use Mock when it's an object that returns a value that is set to the tested class. I use Stub to mimic an Interface or Abstract class to be tested. In fact, it doesn't really matter what you call it, they are all classes that aren't used in production, and are used as utility classes for testing.
Stub - an object that provides predefined answers to method calls.
Mock - an object on which you set expectations.
Fake - an object with limited capabilities (for the purposes of testing), e.g. a fake web service.
Test Double is the general term for stubs, mocks and fakes. But informally, you'll often hear people simply call them mocks.
I am surprised that this question has been around for so long and nobody has as yet provided an answer based on Roy Osherove's "The Art of Unit Testing".
In "3.1 Introducing stubs" defines a stub as:
A stub is a controllable replacement for an existing dependency
(or collaborator) in the system. By using a stub, you can test your code without
dealing with the dependency directly.
And defines the difference between stubs and mocks as:
The main thing to remember about mocks versus stubs is that mocks are just like stubs, but you assert against the mock object, whereas you do not assert against a stub.
Fake is just the name used for both stubs and mocks. For example when you don't care about the distinction between stubs and mocks.
The way Osherove's distinguishes between stubs and mocks, means that any class used as a fake for testing can be both a stub or a mock. Which it is for a specific test depends entirely on how you write the checks in your test.
When your test checks values in the class under test, or actually anywhere but the fake, the fake was used as a stub. It just provided values for the class under test to use, either directly through values returned by calls on it or indirectly through causing side effects (in some state) as a result of calls on it.
When your test checks values of the fake, it was used as a mock.
Example of a test where class FakeX is used as a stub:
const pleaseReturn5 = 5;
var fake = new FakeX(pleaseReturn5);
var cut = new ClassUnderTest(fake);
cut.SquareIt;
Assert.AreEqual(25, cut.SomeProperty);
The fake instance is used as a stub because the Assert doesn't use fake at all.
Example of a test where test class X is used as a mock:
const pleaseReturn5 = 5;
var fake = new FakeX(pleaseReturn5);
var cut = new ClassUnderTest(fake);
cut.SquareIt;
Assert.AreEqual(25, fake.SomeProperty);
In this case the Assert checks a value on fake, making that fake a mock.
Now, of course these examples are highly contrived, but I see great merit in this distinction. It makes you aware of how you are testing your stuff and where the dependencies of your test are.
I agree with Osherove's that
from a pure maintainability perspective, in my tests using mocks creates more trouble than not using them. That has been my experience, but I’m always learning something new.
Asserting against the fake is something you really want to avoid as it makes your tests highly dependent upon the implementation of a class that isn't the one under test at all. Which means that the tests for class ActualClassUnderTest can start breaking because the implementation for ClassUsedAsMock changed. And that sends up a foul smell to me. Tests for ActualClassUnderTest should preferably only break when ActualClassUnderTest is changed.
I realize that writing asserts against the fake is a common practice, especially when you are a mockist type of TDD subscriber. I guess I am firmly with Martin Fowler in the classicist camp (See Martin Fowler's "Mocks aren't Stubs") and like Osherove avoid interaction testing (which can only be done by asserting against the fake) as much as possible.
For fun reading on why you should avoid mocks as defined here, google for "fowler mockist classicist". You'll find a plethora of opinions.
As mentioned by the top-voted answer, Martin Fowler discusses these distinctions in Mocks Aren't Stubs, and in particular the subheading The Difference Between Mocks and Stubs, so make sure to read that article.
Rather than focusing on how these things are different, I think it's more enlightening to focus on why these are distinct concepts. Each exists for a different purpose.
Fakes
A fake is an implementation that behaves "naturally", but is not "real". These are fuzzy concepts and so different people have different understandings of what makes things a fake.
One example of a fake is an in-memory database (e.g. using sqlite with the :memory: store). You would never use this for production (since the data is not persisted), but it's perfectly adequate as a database to use in a testing environment. It's also much more lightweight than a "real" database.
As another example, perhaps you use some kind of object store (e.g. Amazon S3) in production, but in a test you can simply save objects to files on disk; then your "save to disk" implementation would be a fake. (Or you could even fake the "save to disk" operation by using an in-memory filesystem instead.)
As a third example, imagine an object that provides a cache API; an object that implements the correct interface but that simply performs no caching at all but always returns a cache miss would be a kind of fake.
The purpose of a fake is not to affect the behavior of the system under test, but rather to simplify the implementation of the test (by removing unnecessary or heavyweight dependencies).
Stubs
A stub is an implementation that behaves "unnaturally". It is preconfigured (usually by the test set-up) to respond to specific inputs with specific outputs.
The purpose of a stub is to get your system under test into a specific state. For example, if you are writing a test for some code that interacts with a REST API, you could stub out the REST API with an API that always returns a canned response, or that responds to an API request with a specific error. This way you could write tests that make assertions about how the system reacts to these states; for example, testing the response your users get if the API returns a 404 error.
A stub is usually implemented to only respond to the exact interactions you've told it to respond to. But the key feature that makes something a stub is its purpose: a stub is all about setting up your test case.
Mocks
A mock is similar to a stub, but with verification added in. The purpose of a mock is to make assertions about how your system under test interacted with the dependency.
For example, if you are writing a test for a system that uploads files to a website, you could build a mock that accepts a file and that you can use to assert that the uploaded file was correct. Or, on a smaller scale, it's common to use a mock of an object to verify that the system under test calls specific methods of the mocked object.
Mocks are tied to interaction testing, which is a specific testing methodology. People who prefer to test system state rather than system interactions will use mocks sparingly if at all.
Test doubles
Fakes, stubs, and mocks all belong to the category of test doubles. A test double is any object or system you use in a test instead of something else. Most automated software testing involves the use of test doubles of some kind or another. Some other kinds of test doubles include dummy values, spies, and I/O blackholes.
The thing that you assert on it is called a mock object.
Everything else that just helped the test run is a stub.
To illustrate the usage of stubs and mocks, I would like to also include an example based on Roy Osherove's "The Art of Unit Testing".
Imagine, we have a LogAnalyzer application which has the sole functionality of printing logs. It not only needs to talk to a web service, but if the web service throws an error, LogAnalyzer has to log the error to a different external dependency, sending it by email to the web service administrator.
Here’s the logic we’d like to test inside LogAnalyzer:
if(fileName.Length<8)
{
try
{
service.LogError("Filename too short:" + fileName);
}
catch (Exception e)
{
email.SendEmail("a","subject",e.Message);
}
}
How do you test that LogAnalyzer calls the email service correctly when the web service throws an exception?
Here are the questions we’re faced with:
How can we replace the web service?
How can we simulate an exception from the web service so that we can
test the call to the email service?
How will we know that the email service was called correctly or at
all?
We can deal with the first two questions by using a stub for the web service. To solve the third problem, we can use a mock object for the email service.
A fake is a generic term that can be used to describe either a stub or a mock.In our test, we’ll have two fakes. One will be the email service mock, which we’ll use to verify that the correct parameters were sent to the email service. The other will be a stub that we’ll use to simulate an exception thrown from the web service. It’s a stub because we won’t be using the web service fake to verify the test result, only to make sure the test runs correctly. The email service is a mock because we’ll assert against it that it was called correctly.
[TestFixture]
public class LogAnalyzer2Tests
{
[Test]
public void Analyze_WebServiceThrows_SendsEmail()
{
StubService stubService = new StubService();
stubService.ToThrow= new Exception("fake exception");
MockEmailService mockEmail = new MockEmailService();
LogAnalyzer2 log = new LogAnalyzer2();
log.Service = stubService
log.Email=mockEmail;
string tooShortFileName="abc.ext";
log.Analyze(tooShortFileName);
Assert.AreEqual("a",mockEmail.To); //MOCKING USED
Assert.AreEqual("fake exception",mockEmail.Body); //MOCKING USED
Assert.AreEqual("subject",mockEmail.Subject);
}
}
Unit testing - is an approach of testing where the unit(class, method) is under control.
Test double - is not a primary object(from OOP world). It is a realisation which is created temporary to test, check or during development. And they are created for closing dependencies of tested unit(method, class...)
Test doubles types:
fake object is a real implementation of interface(protocol) or an extend which is using an inheritance or other approaches which can be used to create - is dependency. Usually it is created by developer as a simplest solution to substitute some dependency
stub object is a bare object(0, nil and methods without logic) with extra state which is predefined(by developer) to define returned values. Usually it is created by framework
class StubA: A {
override func foo() -> String {
return "My Stub"
}
}
mock object is very similar to stub object but the extra state is changed during program execution to check if something happened(method was called, arguments, when, how often...).
class MockA: A {
var isFooCalled = false
override func foo() -> String {
isFooCalled = true
return "My Mock"
}
}
spy object is a real object with a "partial mocking". It means that you work with a non-double object except mocked behavior
dummy object is object which is necessary to run a test but no one variable or method of this object is not called.
stub vs mock
Martin Fowler said
There is a difference in that the stub uses state verification while the mock uses behavior verification.
[Mockito mock vs spy]
All of them are called Test Doubles and used to inject the dependencies that your test case needs.
Stub:
It already has a predefined behavior to set your expectation
for example, stub returns only the success case of your API response
A mock is a smarter stub. You verify your test passes through it.
so you could make amock that return either the success or failure success depending on the condition could be changed in your test case.
If you are familiar with Arrange-Act-Assert, then one way of explaining the difference between stub and mock that might be useful for you, is that stubs belong to the arrange section as they are for arranging input state, and mocks belong to the assert section as they are for asserting results against.
Dummies don't do anything. They are just for filling up parameter lists, so that you don't get undefined or null errors. They also exist to satisfy the type checker in statically typed languages, so that you can be allowed to compile and run.
Stub, Fakes and Mocks have different meanings across different sources. I suggest you to introduce your team internal terms and agree upon their meaning.
I think it is important to distinguish between two approaches:
- behaviour validation (implies behaviour substitution)
- end-state validation (implies behaviour emulation)
Consider email sending in case of error. When doing behaviour validation - you check that method Send of IEmailSender was executed once. And you need to emulate return result of this method, return Id of the sent message. So you say: "I expect that Send will be called. And I will just return dummy (or random) Id for any call". This is behaviour validation:
emailSender.Expect(es=>es.Send(anyThing)).Return((subject,body) => "dummyId")
When doing state validation you will need to create TestEmailSender that implements IEmailSender. And implement Send method - by saving input to some data structure that will be used for future state verification like array of some objects SentEmails and then it tests you will check that SentEmails contains expected email. This is state validation:
Assert.AreEqual(1, emailSender.SentEmails.Count)
From my readings I understood that Behaviour validation usually called Mocks.
And State validation usually called Stubs or Fakes.
It's a matter of making the tests expressive. I set expectations on a Mock if I want the test to describe a relationship between two objects. I stub return values if I'm setting up a supporting object to get me to the interesting behaviour in the test.
stub and fake are objects in that they can vary their response based on input parameters. the main difference between them is that a Fake is closer to a real-world implementation than a stub. Stubs contain basically hard-coded responses to an expected request. Let see an example:
public class MyUnitTest {
#Test
public void testConcatenate() {
StubDependency stubDependency = new StubDependency();
int result = stubDependency.toNumber("one", "two");
assertEquals("onetwo", result);
}
}
public class StubDependency() {
public int toNumber(string param) {
if (param == “one”) {
return 1;
}
if (param == “two”) {
return 2;
}
}
}
A mock is a step up from fakes and stubs. Mocks provide the same functionality as stubs but are more complex. They can have rules defined for them that dictate in what order methods on their API must be called. Most mocks can track how many times a method was called and can react based on that information. Mocks generally know the context of each call and can react differently in different situations. Because of this, mocks require some knowledge of the class they are mocking. a stub generally cannot track how many times a method was called or in what order a sequence of methods was called. A mock looks like:
public class MockADependency {
private int ShouldCallTwice;
private boolean ShouldCallAtEnd;
private boolean ShouldCallFirst;
public int StringToInteger(String s) {
if (s == "abc") {
return 1;
}
if (s == "xyz") {
return 2;
}
return 0;
}
public void ShouldCallFirst() {
if ((ShouldCallTwice > 0) || ShouldCallAtEnd)
throw new AssertionException("ShouldCallFirst not first thod called");
ShouldCallFirst = true;
}
public int ShouldCallTwice(string s) {
if (!ShouldCallFirst)
throw new AssertionException("ShouldCallTwice called before ShouldCallFirst");
if (ShouldCallAtEnd)
throw new AssertionException("ShouldCallTwice called after ShouldCallAtEnd");
if (ShouldCallTwice >= 2)
throw new AssertionException("ShouldCallTwice called more than twice");
ShouldCallTwice++;
return StringToInteger(s);
}
public void ShouldCallAtEnd() {
if (!ShouldCallFirst)
throw new AssertionException("ShouldCallAtEnd called before ShouldCallFirst");
if (ShouldCallTwice != 2) throw new AssertionException("ShouldCallTwice not called twice");
ShouldCallAtEnd = true;
}
}
According to the book "Unit Testing Principles, Practices, and Patterns by Vladimir Khorikov" :
Mocks: help to emulate and examine outcoming interactions. These interactions are calls the SUT makes to its dependencies to change their state. In other words it helps to examine the interaction (behaviour) of SUT and its dependencies. mocks could be :
Spy : created manually
Mocks : created using framework
Stubs: helps to emulate incoming interactions. These interactions are calls the SUT makes to its dependencies to get input data. IN other words it helps to test the data passed to SUT. It could be 3 types
Fake: is usually implemented to replace a dependency that doesn’t yet exist.
Dummy: is hard-coded value.
Stubs: Fledged dependency that you configure to return different values for different scenarios.
In xUnit Test Patterns book by Gerard Meszaros There is a nice table that gives a good insight about differences
I tend to use just 2 terms - Fake and Mock.
Mock only when using a mocking framework like Moq for example because it doesn't seem right to refer to it as a Fake when it's being created with new Mock<ISomething>() - while you can technically use a mocking framework to create Stubs or Fakes, it just seems kind of dumb to call it that in this situation - it has to be a Mock.
Fake for everything else. If a Fake can be summarised as an implementation with reduced capabilities, then I think a Stub could also be a Fake (and if not, who cares, everyone knows what I mean, and not once has anyone ever said "I think you'll find that's a Stub")