I read that MbUnit is NUnit on steroids, but I don't see why. From what I have read on here, I hear that NUnit is more popular over MbUnit. One of the main reasons is because it has a fluent interface. Is this the only reason?
Why should I prefer MbUnit over NUnit, or vice-versa?
Even though NUnit now includes the most popular MbUnit advanced features, MbUnit is still more feature-rich, for example:
Contract verifiers
XML assertions
Serialization assertions
External data sources (CSV, XML, etc.)
Parallelizable tests
Fluent interfaces may be nice, but in general they don't add any new features. They just present things to the programmer in a different way.
NUnit is more popular because it was there first (therefore there are more articles about it on the web, and better tooling), and because most programmers don't care about or need the advanced features that MbUnit offers.
NUnit started as a port of JUnit, and has been around a long time. MbUnit came after the fact, and it brought "generative" unit testing. This means it has the ability to take a single unit test and generate several from it. One way to do this is the [RowTest] attribute.
Where a typical unit test will not take any parameters, a RowTest will take parameters and generate multiple tests from that. I believe that NUnit has the concept of RowTest now as well.
[Test]
[Row(1, 1, 2)]
[Row(2, 2, 4)]
[Row(1, 2, 3)]
public void X_plus_Y_equals_Z(x, y, z)
{
Assert.AreEqual(z, x+y);
}
This will result in three tests being run in the test runner. There are also features for rolling back database transactions.
NUnit has the fluent interface for assertions, which is nice, but not really a selling point. NUnit probably also has some better tool support (ReSharper's test runner works with NUnit out of the box, but requires plugins for MbUnit).
In the end, you should pick one framework and go with it. The skills you pick up are very portable from one framework to another.
Just a note for those doing research:
From MbUnit's site: "Gallio and MbUnit are currently on hiatus."
The MbUnit GitHub account shows an extremely sparse commit history of late.
The MbUnit plugin is available only for ReSharper 6 (without any excess manipulations). So if you use Visual Studio 2012 or Visual Studio 2013 you have to use a newer version of ReSharper which supports NUnit out of the box.
Also it looks like MbUnit and Gallio is no longer supported.
Related
I realize there have been a LOT of questions on this topic but somehow I haven't found one that addressed my needs.
My team is looking to start automated Unit testing our application. We have never done it before and nobody on the team has much experience with it. I have been asked to research and find a framework for automating our Unit Testing. So far, I have narrowed the choices down to MSTest, NUnit and xUnit.
All across the internet, I read negative reviews about MSTest but it seems the most convenient tool to me for our purpose.
1) Our application is .NET
2) We use licensed VS IDE for our development across the team.
3) Source Control is Team Foundation Server 2010
4) We plan to integrate CI/CD as part of our software delivery process.
I feel that MSTest integrates right into this setup and provides the cleanest interface for us to work on. Is there a significant advantage with NUnit or xUnit over MSTest that we should consider?
I am biased because I work on NUnit, but the advantage of NUnit or xUnit is that both frameworks offer more functionality like data driven tests, parallel execution and a host of advanced features.
That said, there is nothing wrong with MSTest especially if your team is not very experienced with unit testing. It is well integrated with Visual Studio and with TFS, so adding tests is fairly painless. Both NUnit and xUnit integrate well too, but they may require more setup.
Go ahead and start with MSTest. It is an easy introduction to unit testing. If you start running into some of the limitations of MSTest, then it is a fairly easy migration to NUnit or xUnit. For simple cases, it is just a matter of changing your NuGet packages and then find/replace the attributes.
When it comes to choosing between xUnit and NUnit, both are great choices but each has a different philosophy. NUnit tends to be more flexible and allows for a range of testing styles and can be used for integration tests. It is however flexible enough to allow you to shoot yourself in the foot and allows you to write tests that purists would frown upon. NUnit leaves the choice up to you.
xUnit however tends to be a bit more opinionated and pushes you towards the 'pit of success' with unit testing. In my experience, it is great for green field projects, but it can make testing some poorly architected brown-field projects difficult without refactoring.
In the end, you can't go wrong with any of these test frameworks.
I would like to add one negative aspect of XUnit.
It does not have even a basic documentation, nothing beyond "Get Started". No API, nothing.
Here is a brief Analysis of the 3 Frameworks and ranked by color.
Legend of the Coloring:
MSTest it is good only because it is simple and integrated into Visual Studio and it comes from Microsoft. Its not flexible enough and not very extensible. But more than just a unit test framework, it also supports Integration tests, Web (load) performance tests. The other 2 are pure Unit Test Frameworks. So I would say MSTest is "Jack of all Trades, Master of none."
NUnit is my choice because its been the best consistently long enough. It is more flexible and simple with very good user base, documentation and community support.
XUnit is least of my choice because it follows a slightly different style and personally I do not see much advantage over NUnit. In some cases you will need to do some refactoring your code to fit into the test cases. But this is just my opinion as there are many who prefers XUnit.
I have listed down the comparisons in two categories:
Framework:
Implementation:
The source of these comparisons
The company I'm working at uses xUnit to write Integration tests. xUnit works perfectly for us but we will like to extract more statistical information out of runs. For example - "How many times did this specific Test-Case has been failing in the last month", and maybe even sort it out nicely on a pie chart.
Since we have a Microsoft Test Manager license, I did some research on it, and it seems like it does support more detailed reports. I also like the coupling between Manual Test-Cases to Automation Test-Cases, and the fact you easily identify how much of your Test-Cases are automated.
Sadly enough, Test Manager only support MSTest integration out of the box. I did noticed however that the MSTest.ext alternative - VSTest.exe is able to run xUnit tests, and even output TRX result file. Is there any way to integrate xUnit (or nUnit) to the Test Manager somehow? Has anyone done so in the past? we prefer to use Test Manager, but I'm interesting to know if there is an alternative that support a couple Test-Cases with Automated Test-Cases and a way to get statistical information about multiple-runs.
Thank you.
This tool will allow you to associate NUnit and xUnit test cases with Microsoft Test Manager.
https://github.com/JakeGinnivan/TestCaseAutomationAssigner
After some research on the Microsoft Forum (and personally speaking with Microsoft representative), It seems like it's not possible to use xUnit with MS Test Manager.
We decided not to use MS Test Manager and handle all our test runs using VSTest.exe and xUnit categories.
Edit: It's now possible to use an external tool for that called "TestCaseAutomationAssigner". See Jeff's answer for more information.
I need to write a unit test suite for a project I am developing in my spare time. Being a CL newbie I was overwhelmed by the amount of choices for a CL implementation, I spent quite some time to choose one. Now I am facing exactly the same thing with unit test frameworks.
A quick glance at http://www.cliki.net/test%20framework shows 20 unit test frameworks! Choice is good but for a novice like me this can be a bit confusing and given the number of frameworks it would be painful to try them all.
I would like to use a framework which:
Is reasonably well maintained
Easy to use but with some degree of flexibility
Offers some sort of integration with Emacs (or it is possible to easily integrate it with Emacs)
It is possible to integrate it with git post-commit hooks
It is possible to integrate it with a continous integration system (such as buildbot)
What are your experiences in this field?
Did you see the link to http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.html off the Test framework comparison link on that cliki page you mention? Phil gives his impressions, and what it looks like to use the various test frameworks.
I personally prefer lisp-unit. It's simple to use and has most of the common types of tests.
http://www.cliki.net/lisp-unit
http://repo.or.cz/w/lisp-unit.git/blob_plain/master:/documentation/lisp-unit.html
I don't think it has any integration with post-commit hooks or buildbot built in.
Which do you prefer?
What are the advantages and disadvantages of each for Unit Testing?
EDIT: I will admit that Team System offers a lot more than just Unit Testing, such as performance and load testing of applications and databases. This question was centering around writing unit tests and which do you prefer.
Nunit:
Advantages:
Free
Very similar to team system in attributs and methods for assertion, some names are even the same
Disadvantages:
Tests must be run via console or external application ( this can be seen as an advantage, but not from my point of view).
Team System testing
Advantages:
A part of VS, you can run tests in a test window.
If you run a team system server you can run tests more easily as a part of the automated build
Disadvantages:
Expensive
Still isn't as stable as NUnit
A comparison between team system and Nunit
We use team system 2008 as we are gold certified partners to microsoft, but earlier used Nunit due to bug related issues in VS 2005. I prefer the VS solution.
Both are good solutions for your work, look also out for other free solutions like:
Good alternatives to Team System
One very specific reason, is that NUnit won't tie you to the professional edition of the visual studio.
Update: Here is a link about unit testing support on Professional edition in vs 2008: http://msdn.microsoft.com/en-us/library/bb385902.aspx
One other advantage of NUnit is that it doesn't require that you add anything to your test classes. MSTest requires the presence of a TestContext property. We started out with MSTest but converted to NUnit. I also find NUnit to be significantly faster and I prefer ReSharper's test runner UI.
Currently NUnit has test categories that allow you to run unit tests separately from slower integration tests.
MS Tests has no such built-in mechanism.
When using MS Tests, you can use CHESS:
CHESS is a tool for systematically testing multithreaded code. Given a concurrent test, CHESS systematically drives the test along all possible thread interleavings.
Also, I found a nice comparison here that claims MS Tests are a little slower than NUnit, but I didn't check it myself.
Doesn't Visual Studio 2008 allow you to use other testing frameworks when you create the test project? I vaguely remember this from watching the old MVC Framework videos back when Hanselman was doing the preview 2 or 3 videos.
This would allow you to use any testing framework you like and still be able to use it in your VS2008 IDE.
What about testing private methods.
Team System create automatically shadow accessors using reflections - does NUnit same?
There is always tools like ReSharper and TestDriven.NET. They will let you run tests from Visual Studio
What is currently the best tool for JavaME unit testing? I´ve never really used unit testing before (shame on me!), so learning curve is important. I would appreciate some pros and cons with your answer. :)
I think it will depend on what kind of tests are you planning to do. Will you be using continuous integration. Is running tests on handsets a must.
If tests are more logic/data processing tests, the you can do fine with JUnit. But if you need to use some classes from javax.microedition.*, then the things will become a bit tricky, but not impossible.
Some examples: a test for text wrapping on screen, that would need javax.microedition.lcdui.Font. You can't just crate a font class using jars shipped with WTK, because at initialization it will be calling some native methods, that are not available.
For these kind of tests I have created a stub implementation of J2ME. Basically these are my own interpretation of J2ME classes. I can set some preconditions there (for example every character is 5 pixels wide, etc). And it is working really great, because my test only need to know, how J2ME classes respond, not how they are internally implemented.
For networking tests I have used MicroEmulator's networking implementation, and it has also worked out well.
One more issue with unit tests - it is better to have your mobile projects as a java project using Java 4,5,6, because writing test in 1.3 is, at leas for me, a pain in the...
I belive, that starting with JUnit will be just fine, to get up and running. And if some other requirements come up (running tests on handsets), then You can explore alternatives.
I'll be honest, the only unit tester I've used in Java is JUnit and a child project for it named DBUnit for database testing... which I'm assuming you won't need under J2ME.
JUnit's pretty easy to use for unit testing, as long as your IDE supports it (Eclipse has support for JUnit built in). You just mark tests with the #Test annotation (org.junit.Test I think). You can also specify methods that should be run #Before or #After each test, as well as before or after the entire class (with #BeforeClass and #AfterClass).
I've never used JUnit under J2ME, though... I work with J2EE at work.
Never found an outstanding one. You can try to read this document :
how to use it
and here the link to : download it
Made by sony ericsson but work for any J2ME development.
And I would recommend you spend some time learning unit testing in plain Java before attacking unit testing on the mobile platform. This may be a to big to swallow one shoot.
Good luck!
There's a framework called J2MEUnit that you could give a try, but it doesn't look like it's still being actively developed:
http://j2meunit.sourceforge.net