What is the Kotlin philosophy of test doubles? With all classes and functions being sealed by default (not open), mocking frameworks like Mockito are clearly not first-class citizens.
A lot has already been written about the problem, and also about possible solutions, but what do the designers of Kotlin have in mind for testing with doubles? Is it fakes and stubs, or should you role your own spies and mocks?
Frameworks like Mockito are first class citizens, including to the point where they are directly mentioned as one of the reasons for the all-open compiler plugin released in Kotlin 1.0.6. Your references to problems/solutions are out of date and it would be nice if those authors updated their posts.
From that release announcement you will see Mockito called out specifically:
The all-open compiler plugin makes classes annotated with a specific annotation and their members open without the explicit open keyword, so it becomes much easier to use frameworks/libraries such as Spring AOP or Mockito. You can read the detailed information about all-open in the corresponding KEEP.
We provide all-open plugin support both for Gradle and Maven, as well as the IDE integration.
The plugin is available for use using the command-line compiler, IDE compiler, and from Gradle and Maven.
Other references:
Using Mockito with Kotlin
A great part of Kotlin's design is with Joshua Bloch's Effective Java in mind, meaning closed by default, immutability, and so on.
Therefore, these problems exist in Java as well. The solution to these problems was and is to use interfaces where possible, so you can provide test doubles, or any other implementation to your production code for that matter.
For tests, these might be mocks, or fakes, or stubs, or what have you. Bear in mind that a well-written in-memory repository for example is way easier to handle in tests than having to set up a mock for that class.
In addition #nhaarman's comments, another option is Mockito-Kotlin. I like that Kotlin's classes are final by default, so the all-open compiler plugin isn't a good option for me.
Related
I have for long worked on EasyMocking in JUnits. I am pretty much comfortable with this but now I want to know how EasyMocks are different from Jmockits. I tried going through their documentation and I found out that the syntax is a bit different. But yet I could not figure out if there is any difference in their performances.
Can anyone help me figure out what are the points that make either of them better than the other? Is there any special element in JMockit that is not found in the other?
Thanks in advance...
There are many differences between JMockit and EasyMock/jMock/Mockito/PowerMock.
These are the major ones:
Support for integration testing: JMockit supports an out-of-container integration testing approach, similar to what the Spring Test module provides, but also supporting Java EE. The other mocking libraries only support isolated unit testing with mock objects.
A "faking" API (see also in xUnit Patterns), in addition to the mocking API. Each one of the other mocking libraries only have a mocking API.
Full support for "mocking", in addition to "mock objects". Other mocking libraries work with mock objects which they create and which need to be passed in somehow to the code under test. With EasyMock/jMock/Mockito, static methods, constructors, and "new-ed" objects cannot be mocked at all. PowerMock supports full mocking as well, but still focused on mock objects (specifically, new-ed objects must be "replaced" with mock objects through whenNew recording, while with JMockit a test can simply declare a #Mocked field).
Support for mocking/faking of final classes and methods. Only PowerMock also provides this support. Mockito recently added an "inline mock maker" which adds support for finals, but it's not active by default and may not be as reliable.
Support for mocking/faking of unspecificied subclasses and interface implementations (where the test only declares a base type to be mocked/faked).
In the mocking API, expectations on methods with multiple parameters can be recorded/verified with argument matchers (such as anyString, etc.) only for some parameters, while other mocking APIs require such matchers for every single parameter.
Also in the mocking API, expectations can be explicitly verified after the tested code was exercised, just like in Mockito. EasyMock/jMock do not support this verification model.
As for performance, mocking a type with JMockit (done through class redefinition) probably has a higher runtime overhead when compared to creating a mock object with EasyMock/jMock/Mockito (done through subclass definition), and lower when compared with PowerMock (done through class definition on a custom classloader). However, you should only notice the difference in performance if there is a lot of mocking being done, which most likely indicates overuse of mocking.
You can find a blog post about the differences between them (and also mockito!) here: http://www.baeldung.com/mockito-vs-easymock-vs-jmockit
While they all have different syntax and different ways of working, you should be able to achieve whatever you need in regards to mocking with either framework.
A bit confused on the differences of Mock frameworks like NMock vs the VS 2011 Fakes Framework.
Going through MSDN, what I understand is that Fakes allow you to mock your dependencies just like RhinoMock or NMock, however the approach is different, Fakes generates code to achive this functionality but Mocks framework does not. So is my understanding correct? Is Fakes just another Mock framework
Your question was about how the MS Fakes framework is different from NMock and it appears the other answers have resolved some of that, but here is some more information regarding how they are the same and how they are different. NMock is also similar to RhinoMocks and Moq, so I'm grouping them in with NMock.
There are 3 major differences I see right off between NMock/RhinoMocks/Moq and the MS Fakes Framework:
The MS fakes framework uses generated code, much like Accessors in prior versions of Visual Studio instead of generic types. When you want to use the fakes framework for a dependency, you add the assembly that contains the dependency to the references of the test project and then right-click on it to generate the test doubles (stubs or shims). Then when you are testing, you are actually using these generated classes instead. NMock uses generics to accomplish the same thing (i.e. IStudentRepository studentRepository = mocks.NewMock<IStudentRepository>()). In my opinion, the MS Fakes framework approach inhibits code navigation and refactoring from within the tests since you are actually working against a generated class, not your real interface.
The MS fakes framework supplies stubs and moles (shims), whereas NMock, RhinoMocks, and Moq all provide stubs and mocks. I really don't understand MS's decision to not include mocks and I am, personally not a fan of moles for reasons described below.
With the MS fakes framework, you supply an alternative implementation of the methods you want to stub. Within these alternate implementations, you can specify the return values and track information about how or if the method was called. With NMock, RhinoMocks and Moq, you generate a mock object and then use that object to specify stubbed return values or to track interactions (whether and how the methods were called). I find the MS fakes approach more complex and less expressive.
To clarify the difference in what the frameworks provide: NMock, RhinoMocks and Moq all provide two types of test doubles (stubs and mocks). The fakes framework provides stubs and moles (they call them shims), and unfortunately does not include mocks. In order to understand the differences and similarities between NMock and MS Fakes, it is helpful to understand what these different types of test doubles are:
Stubs: Stubs are used when you need to provide a values for methods or properties that will be asked of your test doubles by the method under test. For example, when my method under test calls the DoesStudentExist() method of the IStudentRepository test double, I want it to return true.
The idea of stubs in NMock and MS fakes is the same, but with NMock you would do something like this:
Stub.On(mockStudentRepository).Method("DoesStudentExist").Will(Return.Value(true));
And with MSFakes you would do somethign like this:
IStudentRepository studentRepository = new DataAccess.Fakes.StubIStudentRepository() // Generated by Fakes.
{
DoesStudentExistInt32 = (studentId) => { return new Student(); }
};
Notice in the MS Fakes example you create an entirely new implementation for the DoesStudentExist method (Note that it is called DoesStudentExistInt32 because the fakes framework appends the parameter data types to the method names when it generates the stub objects, I think this obscures the clarity of the tests). To be honest the NMock implementation also bugs me because it uses a string to identify the method name. (Forgive me if I've misunderstood how NMock is intended to be used.) This approach really inhibits refactoring and I'd highly recommend RhinoMocks or Moq over NMock for this reason.
Mocks: Mocks are used to verify interaction between your method under test and its dependencies. With NMock, you do this by setting expectations similar to this:
Expect.Once.On(mockStudentRepository).Method("Find").With(123);
This is another reason why I'd prefer RhinoMocks and Moq over NMock, NMock uses the older expectation style whereas RhinoMocks and Moq both support the Arrange/Act/Assert approach where you specify you expected interactions as assertions at the end of the test like this:
stubStudentRepository.AssertWasCalled( x => x.Find(123));
Again, note that RhinoMocks uses a lambda instead of a string to identify the method. The ms fakes framework does not provide mocks at all. This means that in your stubbed out implementations (see the description of stubs above) you have to set variables that you later verify were set correctly. That would look something like this:
bool wasFindCalled = false;
IStudentRepository studentRepository = new DataAccess.Fakes.StubIStudentRepository()
{
DoesStudentExistInt32 = (studentId) =>
{
wasFindCalled = true;
return new Student();
}
};
classUnderTest.MethodUnderTest();
Assert.IsTrue(wasFindCalled);
I find this approach to be a little convoluted since you have to track the call up in the stub and then assert it later in the test. I find the NMock, and especially the RhinoMocks, examples to be more expressive.
Moles (Shims): To be frank, I do not like moles, because of their potential for misuse. One of the things I like so much about unit testing (and TDD in particular) is that testing your code helps you to understand where you have written poor code. This is because testing poorly written code is difficult. This is not true when using moles because moles are actually designed to allow you to test against dependencies that are not injected or to test private methods. They work similarly to stubs, except that you use a ShimsContext like this:
using (ShimsContext.Create())
{
System.Fakes.ShimDateTime.NowGet = () => { return new DateTime(fixedYear, 1, 1); };
}
My worry with shims is that people will start seeing them as "an easier way to unit test" because it doesn't force you to write code the way you should. For a more complete write-up on this concept see this post of mine:
https://jcoop.io/2012/05/21/solid-code-for-solid-reasons/
For more information on some concerns related to the fakes frameworks take a look at these posts:
https://jcoop.io/2012/03/16/38/
If you're interested in learning RhinoMocks here's a Pluralsight training video (full disclosure - I wrote this course and get paid royalties for views, but I think it applies to this discussion so I'm including it here):
http://www.pluralsight.com/training/Courses/TableOfContents/rhinomock-fundamentals
You are correct, but there's more to the story. The most important things to take away from this answer are:
Your architecture should make proper use of stubs and dependency injection, rather than relying on the crutch of Fakes and mocks
Fakes and mocks are useful for testing code you shouldn't or can't change, such as:
Legacy code that does not make use (or efficient use) of stubs
3rd party APIs
Resources for which you have no source code
Shims (known as "Moles", during development) is indeed a mocking framework that operates by way of detouring calls. Instead of painstakingly building a mock (yes, even using Moq is relatively painful!), shims simply use the production code object already in place. Shims simply re-route the call from the production target to the test delegate.
Stubs are generated from interfaces in the target project. The Stub object is just that -- an implementation of the interface. The benefit to using the Stub type is that you can quickly generate a stub without cluttering up your test project with many one-time use stubs, not to mention wasting time creating them. Of course, you should still create concrete stubs, for use across many tests.
Efficiently implementing Fakes (Shims, Mocks and Stub types) takes a little getting used to, but is well worth the effort. I have personally saved weeks of development time, through use of the Shims/Mole, Mocks, and Stub types. I hope you have as much fun with the technology as I have!
As I understand it, the Visual Studio team wanted to avoid competing with the various mock libraries available for .NET. MS often faces hard decisions like this. They are dammed if they don't provide certain functionality ("why doesn't MS provide us with a mock library; mocks are such a common requirement?") and damned if they do ("why is Microsoft acting so aggressively and driving its natural supporters out of the market?") Very often, but not always, they decide to hold back from simply providing their own alternative to available and well-received technologies. That seems to be the case here.
The shim feature of Fakes is really, really useful. Sure, there are dangers. It takes some discipline to ensure you only use this where necessary. However, it fills a big gap. My main complaint is that it is only delivered with the Ultimate edition of VS 2012 and therefore will only be available to a subsection of the .NET development community. What a pity.
Fakes includes two different kinds of "fake" object. The first, called a "stub", is essentially an auto-generated dummy whose default behaviour can (and usually would) be overridden to make it a more interesting mock. It does, however, lack some of the features that most of the currently available mocking frameworks offer. For example, if you want to check that a method on a stub instance was invoked, you would need to add the logic for this yourself. Basically, if you're authoring your own mocks manually now, stubs would probably seem like an improvement. However, if you're already using a more full-featured mocking framework, you might feel like there are some important pieces missing from Fakes stubs.
The other category of object offered by Fakes, called a "shim", exposes a mechanism for replacing behaviour of dependencies that have not been (or cannot be) decoupled adequately for standard replacement via mocks. AFAIK, TypeMock is the only one of the major mocking frameworks that currently offers this sort of functionality.
BTW, if you have tried out Moles before, Fakes is essentially the same thing, finally making its way out of Microsoft Research and into an actual product.
Regarding Fake (Shim + Stub) objects, it has been well defined above, though I guess the last paragraph in the last comment summarizes the whole situation pretty well.
Though a lot of people will argue that Fake (Shim + Stub) objects are good assets to have in some unit testing cases, the downside is that no matter if you're using Visual Studio 2012 or Visual Studio 2013, these options are ONLY available with Premium or Ultimate versions. IOW, this means that you WILL NOT be to run ANY of those Fakes (Shim + Stub) on any Pro version.
You may probably see the Fakes (Shim + Stub) menu option on Pro versions, but beware, there are some pretty strong chances that you will end up with absolutely nothing... It won't generate any compilation error telling you that something important is missing, options are just not there, so don't waste your time...
It's an important factor to consider in a dev team, especially if one is the only one using Ultimate version while everybody else uses Pro version... Moq on the other hand can easily be installed through Nuget no matter which Visual Studio version you use. I had no problem using Moq, the key with any tool is to know what they're used for and how to use them properly ;)
I'm moderately new to test frameworks and I have been reading up on Moq and this introductory post used a way of organising tests that I had not seen before. Further research shows that we in the .Net world tend to meld the terms BDD and Context/Specification (CS) Testing. I don't want to get into that argument - I am primarily interested in achieving this style of writing test classes.
This article shows the approach again and makes explicit the use of a base class that allows us to construct our specification through the test framework.
This issue I have is that I cannot see an instance method under MSTest that would allow a test fixture to be initialised just once for each test. The best I can see is the constructor of the test class but that feels a bit wrong. In NUnit one could use [TestFixtureSetup]. Is there an equivalent using Visual Studio's built in test framework?
Edit
I've subsequently moved to NUnit which provides the flexibility I require.
After some investigation I have moved to using NUnit. There are several reasons for preferring NUnit over MSTest but, with regards to this particular issue, the compelling reason is that the only class-wide initialisation method supported by MSTest (that doesn't run for every test) is for static methods which is not what I'm looking for.
As Jason points out [ClassInitialise] will provide this static, class wide initialisation in MSTest. There isn't really a way of mimicking the behavior of [TextFixtureSetup] found in NUnit such that an instance method is run once before any test in the fixture is run.
To replicate NUnit's [TestFixtureSetUp] for MSTest, try [ClassInitialize]. Here's the link where I found that information. This is another useful blog article that might help you.
I am just starting to do Test Driven Development, and I am wondering the major differences between RhinoMock, TypeMock, and NUnit's built-in mocking?
Any information would be greatly appreciated!
TypeMock is a commercial product (meaning you'll have to pay for it) but will allow you to mock concrete objects - unlike RhinoMocks/NUnit/MoQ which can only mock an interface/abstract class. How it achieves this is borderline black magic, but it does some very clever things with the CLR.
This can be particularly useful when you use libraries in your project that don't use many interfaces. So you could, for example, use TypeMock to mock out a LINQtoSQL datacontext, or Sharepoint objects. However, if you are using TypeMock this is no excuse for bad design in your application.
As far as I'm aware, aside from minor syntax differences, most of the mocking frameworks have moved away from the old record/playback model. Commonly, you set up your mocks by writing expectations using a Fluent Interface.
Personally, I have only used MoQ and I <3 it.
A video called TDD - Understanding Mock Objects by Roy Osherove is very helpful in learning the differences of the different mocking libraries. He doesn't go in great detail of every aspect, but enough for you to understand. I hope this helps. Roy is also the Chief Architect for TypeMock and is a very influential figure in the unit testing arena. I couldn't recommend this video enough for someone who wants to learn how to use mocking and also learn about the library's available.
The main difference between TypeMock and the open-source library's is that TypeMock uses the Profiler API provided by Microsoft instead of a dynamic proxy. This allows TypeMock to mock concrete classes and static methods. In case you aren't sure what the profiler is, it is the same API that is used by tools like JetBrain's dotTrace and RedGate's Ants .Net profilers. TypeMock just uses the API in a different way to fake(mock) what you tell it to.
#RichardOD, thanks for the reminder, his book "The Art of Unit Testing" goes into greater detail where the video doesn't. I own the book and it is very informative.
Rhino.Mocks is an open source, continuously developed and improving framework by one of the industry's most prolific developers. It has been around for a while and hence supports quite a few different paradigms for mocking. It can be a little tougher to learn therefore in the sense that you might find tutorials for the "old" way of doing things. Here's a tip, SetUpResultFor() and Expect.Call() are the old ways of doing things. The new way is mockObject.AssertWasCalled().
I have not had any personal experience with these others but...
MOQ is an open source, continuously developed and improving framework by one of the industry's somewhat less prolific developers (as compared to Ayende). It is newer and therefore lacks some features that Rhino.Mocks has. This is usually not a problem since these features tend to be ones that are somewhat deprecated in Rhino. I have heard that because of this it is slightly easier to learn (mocking frameworks aren't hard to learn by the way).
NUnit Mocks is very quaint as far as mocking goes. It doesn't support the currently preferred Arrange-Act-Assert syntax relying instead on Expect-Verify (record/replay). It also relies on strings to identify method and property names instead of lambdas. This makes it significantly resistant to refactoring. This is a serious problem. I would not recommend it.
TypeMock Isolater is a hardcore for-pay mocking framework from a company (owned by?) Roy Osherove - a guy who knows his testing but also has some mildly controversial opinions as to how to apply it. It is really intense as far as what it can do - getting down to the low level and modifying how CLR objects work. The philosophy behind TypeMock isn't really 100% TDD however. Part of the benefits of TDD is that by embracing the limitations of Mocking frameworks you will design better code. TypeMock blasts those limitations to pieces. As far as I know it is mostly used by people who are trying to get code they have no control over under test.
I use TypeMock all the time and have found it to be a very powerful tool that can improve the coverage of my unit tests. This is because I work with SharePoint and only TypeMock can allow me to mock out SharePoint classes - since they are concrete classes and not interfaces.
Mocking SharePoint classes is not possible with RhinoMock, Moq, NUNit, etc since (I believe) they require interfaces to mock objects, rather then being able to mock the actual concrete classes.
If your code does use a lot of interfaces, and you don't require mocking concrete classes then TypeMock is a bit pricey, but for the power you get, it's worth it.
Mocking sealed classes can be quite a pain. I currently favor an Adapter pattern to handle this, but something about just keeps feels weird.
So, What is the best way you mock sealed classes?
Java answers are more than welcome. In fact, I would anticipate that the Java community has been dealing with this longer and has a great deal to offer.
But here are some of the .NET opinions:
Why Duck Typing Matters for C#
Develoepers
Creating wrappers
for sealed and other types for
mocking
Unit tests for WCF (and Moq)
For .NET, you could use something like TypeMock, which uses the profiling API and allows you to hook into calls to nearly anything.
My general rule of thumb is that objects that I need to mock should have a common interface too. I think this is right design-wise and makes tests a lot easier (and is usually what you get if you do TDD). More about this can be read in the Google Testing Blog latest post (See point 9).
Also, I've been working mainly in Java in the past 4 years and I can say that I can count on one hand the number of times I've created a final (sealed) class. Another rule here is I should always have a good reason to seal a class, as opposed to sealing it by default.
I believe that Moles, from Microsoft Research, allows you to do that. From the Moles page:
Moles may be used to detour any .NET
method, including non-virtual/static
methods in sealed types.
UPDATE: there is a new framework called "Fakes" in the upcoming VS 11 release that is designed to replace Moles:
The Fakes Framework in Visual Studio 11 is the next generation of Moles & Stubs, and will eventually replace it. Fakes is different from Moles, however, so moving from Moles to Fakes will require some modifications to your code. A guide for this migration will be available at a later date.
Requirements: Visual Studio 11 Ultimate, .NET 4.5
The problem with TypeMock is that it excuses bad design. Now, I know that it is often someone else's bad design that it's hiding, but permitting it into your development process can lead very easily to permitting your own bad designs.
I think if you're going to use a mocking framework, you should use a traditional one (like Moq) and create an isolation layer around the unmockable thing, and mock the isolation layer instead.
I came across this problem recently and after reading / searching web, seems like there is no easy way around except to use another tool as mentioned above.
Or crude of handling things as I did:
Create instance of sealed class without getting constructor called.
System.Runtime.Serialization.FormatterServices.GetUninitializedObject(instanceType);
Assign values to your properties / fields via reflection
YourObject.GetType().GetProperty("PropertyName").SetValue(dto, newValue, null);
YourObject.GetType().GetField("FieldName").SetValue(dto, newValue);
I almost always avoid having dependencies on external classes deep within my code. Instead, I'd much rather use an adapter/bridge to talk to them. That way, I'm dealing with my semantics, and the pain of translating is isolated in one class.
It also makes it easier to switch my dependencies in the long run.
It is perfectly reasonable to mock a sealed class because many framework classes are sealed.
In my case I'm trying to mock .Net's MessageQueue class so that I can TDD my graceful exception handling logic.
If anyone has ideas on how to overcome Moq's error regarding "Invalid setup on a non-overridable member", please let me know.
code:
[TestMethod]
public void Test()
{
Queue<Message> messages = new Queue<Message>();
Action<Message> sendDelegate = msg => messages.Enqueue(msg);
Func<TimeSpan, MessageQueueTransaction, Message> receiveDelegate =
(v1, v2) =>
{
throw new Exception("Test Exception to simulate a failed queue read.");
};
MessageQueue mockQueue = QueueMonitorHelper.MockQueue(sendDelegate, receiveDelegate).Object;
}
public static Mock<MessageQueue> MockQueue
(Action<Message> sendDelegate, Func<TimeSpan, MessageQueueTransaction, Message> receiveDelegate)
{
Mock<MessageQueue> mockQueue = new Mock<MessageQueue>(MockBehavior.Strict);
Expression<Action<MessageQueue>> sendMock = (msmq) => msmq.Send(It.IsAny<Message>()); //message => messages.Enqueue(message);
mockQueue.Setup(sendMock).Callback<Message>(sendDelegate);
Expression<Func<MessageQueue, Message>> receiveMock = (msmq) => msmq.Receive(It.IsAny<TimeSpan>(), It.IsAny<MessageQueueTransaction>());
mockQueue.Setup(receiveMock).Returns<TimeSpan, MessageQueueTransaction>(receiveDelegate);
return mockQueue;
}
Although it's currently only available in beta release, I think it's worthwhile keeping in mind the shim feature of the new Fakes framework (part of the Visual Studio 11 Beta release).
Shim types provide a mechanism to detour any .NET method to a user defined delegate. Shim types are code-generated by the Fakes generator, and they use delegates, which we call shim types, to specify the new method implementations. Under the hood, shim types use callbacks that were injected at runtime in the method MSIL bodies.
Personally, I was looking at using this to mock the methods on sealed framework classes such as DrawingContext.
I generally take the route of creating an interface and adaptor/proxy class to facilitate mocking of the sealed type. However, I've also experimented with skipping creation of the interface and making the proxy type non-sealed with virtual methods. This worked well when the proxy is really a natural base class that encapsulates and users part of the sealed class.
When dealing with code that required this adaptation, I got tired of performing the same actions to create the interface and proxy type so I implemented a library to automate the task.
The code is somewhat more sophisticated than the sample given in the article you reference, as it produces an assembly (instead of source code), allows for code generation to be performed on any type, and doesn't require as much configuration.
For more information, please refer to this page.
Is there a way to implement a sealed class from an interface... and mock the interface instead?
Something in me feels that having sealed classes is wrong in the first place, but that's just me :)