I realize there have been a LOT of questions on this topic but somehow I haven't found one that addressed my needs.
My team is looking to start automated Unit testing our application. We have never done it before and nobody on the team has much experience with it. I have been asked to research and find a framework for automating our Unit Testing. So far, I have narrowed the choices down to MSTest, NUnit and xUnit.
All across the internet, I read negative reviews about MSTest but it seems the most convenient tool to me for our purpose.
1) Our application is .NET
2) We use licensed VS IDE for our development across the team.
3) Source Control is Team Foundation Server 2010
4) We plan to integrate CI/CD as part of our software delivery process.
I feel that MSTest integrates right into this setup and provides the cleanest interface for us to work on. Is there a significant advantage with NUnit or xUnit over MSTest that we should consider?
I am biased because I work on NUnit, but the advantage of NUnit or xUnit is that both frameworks offer more functionality like data driven tests, parallel execution and a host of advanced features.
That said, there is nothing wrong with MSTest especially if your team is not very experienced with unit testing. It is well integrated with Visual Studio and with TFS, so adding tests is fairly painless. Both NUnit and xUnit integrate well too, but they may require more setup.
Go ahead and start with MSTest. It is an easy introduction to unit testing. If you start running into some of the limitations of MSTest, then it is a fairly easy migration to NUnit or xUnit. For simple cases, it is just a matter of changing your NuGet packages and then find/replace the attributes.
When it comes to choosing between xUnit and NUnit, both are great choices but each has a different philosophy. NUnit tends to be more flexible and allows for a range of testing styles and can be used for integration tests. It is however flexible enough to allow you to shoot yourself in the foot and allows you to write tests that purists would frown upon. NUnit leaves the choice up to you.
xUnit however tends to be a bit more opinionated and pushes you towards the 'pit of success' with unit testing. In my experience, it is great for green field projects, but it can make testing some poorly architected brown-field projects difficult without refactoring.
In the end, you can't go wrong with any of these test frameworks.
I would like to add one negative aspect of XUnit.
It does not have even a basic documentation, nothing beyond "Get Started". No API, nothing.
Here is a brief Analysis of the 3 Frameworks and ranked by color.
Legend of the Coloring:
MSTest it is good only because it is simple and integrated into Visual Studio and it comes from Microsoft. Its not flexible enough and not very extensible. But more than just a unit test framework, it also supports Integration tests, Web (load) performance tests. The other 2 are pure Unit Test Frameworks. So I would say MSTest is "Jack of all Trades, Master of none."
NUnit is my choice because its been the best consistently long enough. It is more flexible and simple with very good user base, documentation and community support.
XUnit is least of my choice because it follows a slightly different style and personally I do not see much advantage over NUnit. In some cases you will need to do some refactoring your code to fit into the test cases. But this is just my opinion as there are many who prefers XUnit.
I have listed down the comparisons in two categories:
Framework:
Implementation:
The source of these comparisons
Related
The company I'm working at uses xUnit to write Integration tests. xUnit works perfectly for us but we will like to extract more statistical information out of runs. For example - "How many times did this specific Test-Case has been failing in the last month", and maybe even sort it out nicely on a pie chart.
Since we have a Microsoft Test Manager license, I did some research on it, and it seems like it does support more detailed reports. I also like the coupling between Manual Test-Cases to Automation Test-Cases, and the fact you easily identify how much of your Test-Cases are automated.
Sadly enough, Test Manager only support MSTest integration out of the box. I did noticed however that the MSTest.ext alternative - VSTest.exe is able to run xUnit tests, and even output TRX result file. Is there any way to integrate xUnit (or nUnit) to the Test Manager somehow? Has anyone done so in the past? we prefer to use Test Manager, but I'm interesting to know if there is an alternative that support a couple Test-Cases with Automated Test-Cases and a way to get statistical information about multiple-runs.
Thank you.
This tool will allow you to associate NUnit and xUnit test cases with Microsoft Test Manager.
https://github.com/JakeGinnivan/TestCaseAutomationAssigner
After some research on the Microsoft Forum (and personally speaking with Microsoft representative), It seems like it's not possible to use xUnit with MS Test Manager.
We decided not to use MS Test Manager and handle all our test runs using VSTest.exe and xUnit categories.
Edit: It's now possible to use an external tool for that called "TestCaseAutomationAssigner". See Jeff's answer for more information.
I read that MbUnit is NUnit on steroids, but I don't see why. From what I have read on here, I hear that NUnit is more popular over MbUnit. One of the main reasons is because it has a fluent interface. Is this the only reason?
Why should I prefer MbUnit over NUnit, or vice-versa?
Even though NUnit now includes the most popular MbUnit advanced features, MbUnit is still more feature-rich, for example:
Contract verifiers
XML assertions
Serialization assertions
External data sources (CSV, XML, etc.)
Parallelizable tests
Fluent interfaces may be nice, but in general they don't add any new features. They just present things to the programmer in a different way.
NUnit is more popular because it was there first (therefore there are more articles about it on the web, and better tooling), and because most programmers don't care about or need the advanced features that MbUnit offers.
NUnit started as a port of JUnit, and has been around a long time. MbUnit came after the fact, and it brought "generative" unit testing. This means it has the ability to take a single unit test and generate several from it. One way to do this is the [RowTest] attribute.
Where a typical unit test will not take any parameters, a RowTest will take parameters and generate multiple tests from that. I believe that NUnit has the concept of RowTest now as well.
[Test]
[Row(1, 1, 2)]
[Row(2, 2, 4)]
[Row(1, 2, 3)]
public void X_plus_Y_equals_Z(x, y, z)
{
Assert.AreEqual(z, x+y);
}
This will result in three tests being run in the test runner. There are also features for rolling back database transactions.
NUnit has the fluent interface for assertions, which is nice, but not really a selling point. NUnit probably also has some better tool support (ReSharper's test runner works with NUnit out of the box, but requires plugins for MbUnit).
In the end, you should pick one framework and go with it. The skills you pick up are very portable from one framework to another.
Just a note for those doing research:
From MbUnit's site: "Gallio and MbUnit are currently on hiatus."
The MbUnit GitHub account shows an extremely sparse commit history of late.
The MbUnit plugin is available only for ReSharper 6 (without any excess manipulations). So if you use Visual Studio 2012 or Visual Studio 2013 you have to use a newer version of ReSharper which supports NUnit out of the box.
Also it looks like MbUnit and Gallio is no longer supported.
I need to write a unit test suite for a project I am developing in my spare time. Being a CL newbie I was overwhelmed by the amount of choices for a CL implementation, I spent quite some time to choose one. Now I am facing exactly the same thing with unit test frameworks.
A quick glance at http://www.cliki.net/test%20framework shows 20 unit test frameworks! Choice is good but for a novice like me this can be a bit confusing and given the number of frameworks it would be painful to try them all.
I would like to use a framework which:
Is reasonably well maintained
Easy to use but with some degree of flexibility
Offers some sort of integration with Emacs (or it is possible to easily integrate it with Emacs)
It is possible to integrate it with git post-commit hooks
It is possible to integrate it with a continous integration system (such as buildbot)
What are your experiences in this field?
Did you see the link to http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.html off the Test framework comparison link on that cliki page you mention? Phil gives his impressions, and what it looks like to use the various test frameworks.
I personally prefer lisp-unit. It's simple to use and has most of the common types of tests.
http://www.cliki.net/lisp-unit
http://repo.or.cz/w/lisp-unit.git/blob_plain/master:/documentation/lisp-unit.html
I don't think it has any integration with post-commit hooks or buildbot built in.
With the release of BizTalk 2009 we can now use Visual Studio Team Edition 2008 to leverage some of the team test features.
We can unit test maps, schemas, and pipelines ... but from what I can tell, that's about it.
How would you unit test orchestrations for instance?
Well I have been using BizUnit for years already and plan to continue to do so;
What I like about BizUnit is that it is extensible and opensource
So ... Who can give me a few reasons why I should be ditching BizUnit in favor of Team Test in Visual Studio?
You can unit test your orchestration using Bizmock http://www.codeplex.com/bizmock I just looked at it briefly, didn't get a chance to use it on real time projects. But according to the documentation and sample its looks like perfect tool to unit test your orchestration.
I looked at the BizUnit page, and it seems to me that you can use BizUnit in conjunction with Visual Studio Team Test. BizUnit is a declarative testing framework that can augment VS Team Test's capabilities. BizUnit does not require VS Team Test, but VS Team Test can provide an excellent mechanism for driving BizUnit test cases.
The new features of BizTalk 2009 will give you much better support for debugging maps, testing schemas and pipelines but as far as orchestrations go, you still need something like BizUnit to cover that need. I would say the build components alone are a reason to upgrade, so you can setup continuous builds within TFS and the like, which before you had to use an open source solution. So specifically to your question you wouldn't ditch BizUnit for anything and its still needed.
-Bryan
In my experience the answer to how you test BizTalk orchestations depends on the type of application that you are writing in BizTalk.
For example BizUnit works very well for testing the usual integration scenarios, but less so (in my opinion) where the BizTalk solution to be tested is written more like a conventional [middle-tier] application.
In the solution I am developing I have several nested orchestrations invoked from a "parent" orchestration with call-orchestration shape. These child orchestrations have logic/work-flow that can and still need to be tested in isolation.
I have been looking at exercising these from a VS Unit test via a "wrapper" orchestration which accepts the required parameters and invokes the orchestration to be tested (via start or call orchestration shape). The wrapper itself is invoked via an incoming message originating at the unit test.
I am also interested in evaluating bizmock.
Which do you prefer?
What are the advantages and disadvantages of each for Unit Testing?
EDIT: I will admit that Team System offers a lot more than just Unit Testing, such as performance and load testing of applications and databases. This question was centering around writing unit tests and which do you prefer.
Nunit:
Advantages:
Free
Very similar to team system in attributs and methods for assertion, some names are even the same
Disadvantages:
Tests must be run via console or external application ( this can be seen as an advantage, but not from my point of view).
Team System testing
Advantages:
A part of VS, you can run tests in a test window.
If you run a team system server you can run tests more easily as a part of the automated build
Disadvantages:
Expensive
Still isn't as stable as NUnit
A comparison between team system and Nunit
We use team system 2008 as we are gold certified partners to microsoft, but earlier used Nunit due to bug related issues in VS 2005. I prefer the VS solution.
Both are good solutions for your work, look also out for other free solutions like:
Good alternatives to Team System
One very specific reason, is that NUnit won't tie you to the professional edition of the visual studio.
Update: Here is a link about unit testing support on Professional edition in vs 2008: http://msdn.microsoft.com/en-us/library/bb385902.aspx
One other advantage of NUnit is that it doesn't require that you add anything to your test classes. MSTest requires the presence of a TestContext property. We started out with MSTest but converted to NUnit. I also find NUnit to be significantly faster and I prefer ReSharper's test runner UI.
Currently NUnit has test categories that allow you to run unit tests separately from slower integration tests.
MS Tests has no such built-in mechanism.
When using MS Tests, you can use CHESS:
CHESS is a tool for systematically testing multithreaded code. Given a concurrent test, CHESS systematically drives the test along all possible thread interleavings.
Also, I found a nice comparison here that claims MS Tests are a little slower than NUnit, but I didn't check it myself.
Doesn't Visual Studio 2008 allow you to use other testing frameworks when you create the test project? I vaguely remember this from watching the old MVC Framework videos back when Hanselman was doing the preview 2 or 3 videos.
This would allow you to use any testing framework you like and still be able to use it in your VS2008 IDE.
What about testing private methods.
Team System create automatically shadow accessors using reflections - does NUnit same?
There is always tools like ReSharper and TestDriven.NET. They will let you run tests from Visual Studio