I'm moderately new to test frameworks and I have been reading up on Moq and this introductory post used a way of organising tests that I had not seen before. Further research shows that we in the .Net world tend to meld the terms BDD and Context/Specification (CS) Testing. I don't want to get into that argument - I am primarily interested in achieving this style of writing test classes.
This article shows the approach again and makes explicit the use of a base class that allows us to construct our specification through the test framework.
This issue I have is that I cannot see an instance method under MSTest that would allow a test fixture to be initialised just once for each test. The best I can see is the constructor of the test class but that feels a bit wrong. In NUnit one could use [TestFixtureSetup]. Is there an equivalent using Visual Studio's built in test framework?
Edit
I've subsequently moved to NUnit which provides the flexibility I require.
After some investigation I have moved to using NUnit. There are several reasons for preferring NUnit over MSTest but, with regards to this particular issue, the compelling reason is that the only class-wide initialisation method supported by MSTest (that doesn't run for every test) is for static methods which is not what I'm looking for.
As Jason points out [ClassInitialise] will provide this static, class wide initialisation in MSTest. There isn't really a way of mimicking the behavior of [TextFixtureSetup] found in NUnit such that an instance method is run once before any test in the fixture is run.
To replicate NUnit's [TestFixtureSetUp] for MSTest, try [ClassInitialize]. Here's the link where I found that information. This is another useful blog article that might help you.
Related
What is the Kotlin philosophy of test doubles? With all classes and functions being sealed by default (not open), mocking frameworks like Mockito are clearly not first-class citizens.
A lot has already been written about the problem, and also about possible solutions, but what do the designers of Kotlin have in mind for testing with doubles? Is it fakes and stubs, or should you role your own spies and mocks?
Frameworks like Mockito are first class citizens, including to the point where they are directly mentioned as one of the reasons for the all-open compiler plugin released in Kotlin 1.0.6. Your references to problems/solutions are out of date and it would be nice if those authors updated their posts.
From that release announcement you will see Mockito called out specifically:
The all-open compiler plugin makes classes annotated with a specific annotation and their members open without the explicit open keyword, so it becomes much easier to use frameworks/libraries such as Spring AOP or Mockito. You can read the detailed information about all-open in the corresponding KEEP.
We provide all-open plugin support both for Gradle and Maven, as well as the IDE integration.
The plugin is available for use using the command-line compiler, IDE compiler, and from Gradle and Maven.
Other references:
Using Mockito with Kotlin
A great part of Kotlin's design is with Joshua Bloch's Effective Java in mind, meaning closed by default, immutability, and so on.
Therefore, these problems exist in Java as well. The solution to these problems was and is to use interfaces where possible, so you can provide test doubles, or any other implementation to your production code for that matter.
For tests, these might be mocks, or fakes, or stubs, or what have you. Bear in mind that a well-written in-memory repository for example is way easier to handle in tests than having to set up a mock for that class.
In addition #nhaarman's comments, another option is Mockito-Kotlin. I like that Kotlin's classes are final by default, so the all-open compiler plugin isn't a good option for me.
I've read a lot of topic about "Singleton is hard for testing". Most of them posted in 2010 and 2011.
And it tried to figured out why Singleton is bad for testing by using PowerMock. But everything worked fine and i still couldn't find what is the devil of Singleton.
Can you give any example of Singleton in testing and point out the problem?
From a .Net perspective, I suppose it's because many mocking frameworks don't easily support the ability provide mocks, given how the Singleton pattern is commonly implemented, i.e. with static fields, no constructor and a read-only property. Using this typical pattern provides little for external code to hook into.
However, you can use the 'Ambient Context' pattern to achieve the same effect and support 'testability'.
Another interesting issue, depending on your testing framework, is that the singleton might maintain state across tests.
This makes it hard to guarantee that the symptoms seen in one set of unit tests are due to that set of tests, and not the previously run tests. Or if you have a testing framework that randomly re-orders tests, you might find that the behaviour of the tests changes between runs.
I have an existing VS 2005 Std .NET Compact Framework application that I want to do some major refactorings on. Currently there is no unit testing in place, but I want to add this before messing with the code. I have no practical experience with unit testing, even though I know the theory (just never got around actually implementing it; I know: shame on me :-))
Here are some questions I am pondering at the moment:
a) As a beginner, should I use NUnit or NUnitLite (which claims to be easier to use)?
b) Should I aim for running the tests on the mobile device or on the desktop (except for device-specific code of course)? Currently the desktop looks more appealing, especially for including the tests in automated builds...
c) How is the class that I want to test usually included in the test project? My application is a .EXE file, i.e. I can not just reference it like a .DLL assembly from the test project (or can I? Never tried this ...). I checked various NUnit tutorials but either found no mention of that, or one tutorial suggested to copy and paste the class that I want to test into the test project (yuk!). Should I link to the original source code file in my test project? What about private methods, or dependencies on other classes?
d) Should I start modifying my original code to allow better testability, e.g. make private methods public, decouple etc.? This is a bit like refactoring before being able to test, which does not sound good to me ... Or is it better practice to not touch the original code at all in the beginning, even if this means less code coverage etc.?
e) Should I look into any other tools or addons that most people use?
Thanks in advance for any answers (I also appreciate answers if they are only to one or some of the above items).
First, I would recommend you a good book on unit testing: Pragmatic Unit Testing in C#.
It will introduce you to NUnit, but what's more important, the author will provide you a lot of advices how to write good unit tests in general. The xUnit test frameworks are not very complex and you'll get used to their API/workflow very quick. Challenging is the actual process of identifying boundary conditions, decrease coupling and design for testability. It's available as an eBook (PDF) or a printed copy.
Regarding your actual questions (the book will give you some answers, too):
#a) I've no experience with NUnit lite, thus I cannot give you any advice on this point.
#b) Unit tests are supposed to be very local in regard of their dependencies. You aim to test classes independent of each other, thus there would be no need to deploy it on a mobile device first. You won't run the full app, just test components in isolation. Hence, I would recommend to use your desktop machine as the target for your unit test environment. You'll get better turn-around times, too.
#c) You have to reference the assembly that contains the classes you want to test in your test project. The test project will be an assembly itself (DLL). A test runner executes this assembly and uses the stored meta information to run the contained test cases.
#d) It depends a lot on the state and design of your software. But in general I would use a divide and conquer strategy: Introduce interfaces between classes and start to refactor step by step. Write unit tests before you start to change the implementation. The interfaces keep the contracts up and running, but you can change the underlying implementation if necessary. Don't make private methods public just to make them testable. Private methods are internal helpers of the class that support public methods in doing their job. Since you test your public methods you'll assert that your private methods do the right thing.
#e) A helpful add in for Visual Studio is TestDriven.Net. It allows you to run NUnit tests directly from the IDE without changing to NUnit's GUI or console runner.
#c) I haven't tried it, but I would think VisualStudio would let you add a project reference from your test assembly to your actual code assembly, even if its an exe.
As for private methods and such, generally I don't test private methods. In theory, all the private stuff should be used by public or internal methods, so testing those should indirectly test the privates.
I do test public and internals though. One thing I find very helpful is the InternalsVisibleTo attribute:
[assembly:InternalsVisibleTo("MyTestAssembly")]
which you can use to make the internals of one assembly visible to another. I use this attribute to expose the internals to my test assembly, so that it can directly reference them.
It is good practice to use the main method to test a java/.net class?
I've seen it reccommended in some text books, but to me it seems like using a unit testing framework would make more sense...
The main method gives you one point of entry to the class and you can test one aspect of the classes functionality. You could I guess test many but it doesn't seem to make sense as much as using Junit or Nunit.
One obvious advantage seems to be that you can whitebox test the class. That is, you can test the internals of it (private methods for example). You can't do that with unit-tests, nor would you wan't that, they are primarily there to test the interface and the behavior from the users perspective.
I think it could be useful to develop integration tests that are invoked from a main method -- like a test runner -- that tests suites of integration tests. I wouldn't do unit testing this way as unit testing frameworks provide a much better mechanism to do this.
[EDIT] To clarify, I'm not suggesting that each class have static main method to be used for integration tests, but rather that you could write an integration test program with a static main method that would run your suite of integration tests.
The main method can be useful for certain situations, but using a debugger and then writing a unit test (to provide some insurance against regressions) is a more robust solution.
In Java it's accepted to have multiple main methods and use them for testing however .NET doesn't allow this, if you have two mains in the same program you will get compiler error CS0017, and tells you to Compile with /main to specify the type that contains the entry point.
I've got to say the Java way makes more sense to me.
Recently there has been quite some hype around all the different mocking frameworks in the .NET world. I still haven't quite grasped what is so great about them. It doesn't seem to be to hard to write the mocking objects I need myself. Especially with the help of Visual Studio I quickly can write a class that implements the interface I want to mock (it auto-generates almost everything for me) and then write an implementation for the method(s) I need for my test. Done! Why going through the hassle of understanding a mocking framework for the sole purpose of saving a few lines of code. Or is a mocking framework not only about saving lines of code?
Once I finally got the hang of mock objects, I realized that they're essential for unit testing for the same reason that double blind testing or control groups are essential for scientific trials: they isolate what you're actually testing.
If you're testing a class which has quite a bit of interaction via other interfaces, you not only save the lines of code on having to mock each and every interface, but you also gain the ability to do things like "throw an exception if an unexpected method is called" or "exception if these methods are called out of order". You can get remarkably sophisticated with mock frameworks, and though I'll quickly admit there's a large learning curve, when you get up to speed they'll help make your unit tests more thorough without being bloated.
You actually identified one of the key points of a mock framework in your question. The fact that you code the mocks yourself is not something the developer should be concerned with. The mocking frameworks give you implementations of interfaces programatically, plus they are functional (based on your setup of the mock).
What do you do if you are testing an ICustomerDAO, for example, and you want to test some method 14 times each with different outcomes? Implement 14 different classes manually? I doubt that anyone would want to do that.
Mocks give you the power to define what will happen with parts of your classes when you are not concerned with whether or not they will actually work, like throwing exceptions whenever you want them to, returning zero results and making sure you handle that correctly, etc...
They are a great unit testing tool.
Previous questions that may help:
What is a mock and when should you use it?
Mockist vs classical TDD
I find that using a mocking framework allows me to generate tests a lot faster and with better verification that what I expect to happen in the test actually is happengin. I have in the past implemented stubs or fakes myself. I found that I needed to generate stubs specific to the test that I wanted and this took a lot of time. I can create the same test much faster using a mocking framework. The good ones support the generation of fakes, stubs or mocks with straightforward syntax.
It takes a while to get the hang of it, I avoided it for a while but now wouldn't try to work without a mocking framework for the reasons #Chamelaeon states.
Roy Osherove had a poll about Mock Frameworks and down in the comment section, there is a discussion (albeit brief) about whether one needs a Mock Framework or not.
I personally have been manually doing exactly as you stated and it has worked well enough, but this has mainly been out of habit rather than a closely-held opinion on mock frameworks in general.
Well I certainly don't think that you NEED a mocking framework. It's a framework like any other, and it's ultimately designed to save you some time and effort. You can also do things like roll your own common data structures like stacks and queues, but isn't it generally easier to just use the ones built into the class libraries that ship with the compiler/IDE of your language of choice?
I'm sure there are other compelling reasons for using mocking frameworks, though I'd leave it to the TDD and unit testing gurus to answer.
For the same reason you wouldn't try to write unit tests without NUnit. A mocking framework will assist you in verifying state and behavior over hundreds of unit tests. It's worth the 2 weeks or so of pain to get up to speed and really helps you focus on what needs to be tested.
One thing that troubles me about a mocking framework is that "what a function should o/p given an i/p" via
when(mock.someMethod("some arg")).thenReturn("something");
statement is spread across many unit test classes.
Let me elaborate with an example. Lets say there was a DAO Interface function getEmp(int EmpID) which was returning an Employee Object when passed an Employee ID as a parameter. Assume that this function was being mocked by 10 different unit test classes. Now if in the future, this function were changed to return a newer version of the Employee Object, one would have to go to each of the 10 different classes to update this change.
The disadvantages are as follows...
a) I don't know how to figure out all the classes which mock this function so that I can go update this change.
b) My existing test cases which consumes the mock DAO object continue to be blissfully unaware of the changes that have happened to the DAO Interface because the mock has not changed and hence continue to be green.
Ideally, if I were to have coded a single mock class myself and consumed it everywhere, I would have just one place to update for the newer version of the Employee object. Also, once I update at this one place, all my existing test cases which consume the mock would break and I would then know exactly what places I need to go and do an update for the new Employee Object.
Any thoughts on my views..
One of the good things about a mocking framework is that it allows setting expectations on the objects being mocked. With the expectations I can then set up all sorts of conditions to exercise the code thats being tested.
An isolation framework or mocking framework allows you to test the code you want, without its dependencies. It makes for short running tests, allows you to debug quickly, and easily build a safety net of tests around the code. Different frameworks have different features, and as said before - it's a tool, and you should select the right tool for the job.
I've use rhino mocks for a mocking framework. I and 5 other developers used it on a large enterprise application that was an 8 month project. We used tdd on the project. Was it worth it? I guess. Was there such a massive huge selling point to using mocks that I have to use it on every project? In my opinion, no. It is not something that is necessary, it is just a tool that you can use if you want to try it out. Some projects you can roll out your own mock classes as some here say they prefer - it is easier. Other projects are larger and may require a mocking framework. The key word (in my opinion) is MAY require... how much code coverage do you require? To me, that is another consideration to using mocks. The project I did with tdd/rhino mocks we were required to have 80% code coverage so the mocks helped us attain that. If our code coverage requirements were less, for example 40%, we probably would have not used a mocking framework and just wrote our own mock classes as others mention they do.