Which do you prefer?
What are the advantages and disadvantages of each for Unit Testing?
EDIT: I will admit that Team System offers a lot more than just Unit Testing, such as performance and load testing of applications and databases. This question was centering around writing unit tests and which do you prefer.
Nunit:
Advantages:
Free
Very similar to team system in attributs and methods for assertion, some names are even the same
Disadvantages:
Tests must be run via console or external application ( this can be seen as an advantage, but not from my point of view).
Team System testing
Advantages:
A part of VS, you can run tests in a test window.
If you run a team system server you can run tests more easily as a part of the automated build
Disadvantages:
Expensive
Still isn't as stable as NUnit
A comparison between team system and Nunit
We use team system 2008 as we are gold certified partners to microsoft, but earlier used Nunit due to bug related issues in VS 2005. I prefer the VS solution.
Both are good solutions for your work, look also out for other free solutions like:
Good alternatives to Team System
One very specific reason, is that NUnit won't tie you to the professional edition of the visual studio.
Update: Here is a link about unit testing support on Professional edition in vs 2008: http://msdn.microsoft.com/en-us/library/bb385902.aspx
One other advantage of NUnit is that it doesn't require that you add anything to your test classes. MSTest requires the presence of a TestContext property. We started out with MSTest but converted to NUnit. I also find NUnit to be significantly faster and I prefer ReSharper's test runner UI.
Currently NUnit has test categories that allow you to run unit tests separately from slower integration tests.
MS Tests has no such built-in mechanism.
When using MS Tests, you can use CHESS:
CHESS is a tool for systematically testing multithreaded code. Given a concurrent test, CHESS systematically drives the test along all possible thread interleavings.
Also, I found a nice comparison here that claims MS Tests are a little slower than NUnit, but I didn't check it myself.
Doesn't Visual Studio 2008 allow you to use other testing frameworks when you create the test project? I vaguely remember this from watching the old MVC Framework videos back when Hanselman was doing the preview 2 or 3 videos.
This would allow you to use any testing framework you like and still be able to use it in your VS2008 IDE.
What about testing private methods.
Team System create automatically shadow accessors using reflections - does NUnit same?
There is always tools like ReSharper and TestDriven.NET. They will let you run tests from Visual Studio
Related
I realize there have been a LOT of questions on this topic but somehow I haven't found one that addressed my needs.
My team is looking to start automated Unit testing our application. We have never done it before and nobody on the team has much experience with it. I have been asked to research and find a framework for automating our Unit Testing. So far, I have narrowed the choices down to MSTest, NUnit and xUnit.
All across the internet, I read negative reviews about MSTest but it seems the most convenient tool to me for our purpose.
1) Our application is .NET
2) We use licensed VS IDE for our development across the team.
3) Source Control is Team Foundation Server 2010
4) We plan to integrate CI/CD as part of our software delivery process.
I feel that MSTest integrates right into this setup and provides the cleanest interface for us to work on. Is there a significant advantage with NUnit or xUnit over MSTest that we should consider?
I am biased because I work on NUnit, but the advantage of NUnit or xUnit is that both frameworks offer more functionality like data driven tests, parallel execution and a host of advanced features.
That said, there is nothing wrong with MSTest especially if your team is not very experienced with unit testing. It is well integrated with Visual Studio and with TFS, so adding tests is fairly painless. Both NUnit and xUnit integrate well too, but they may require more setup.
Go ahead and start with MSTest. It is an easy introduction to unit testing. If you start running into some of the limitations of MSTest, then it is a fairly easy migration to NUnit or xUnit. For simple cases, it is just a matter of changing your NuGet packages and then find/replace the attributes.
When it comes to choosing between xUnit and NUnit, both are great choices but each has a different philosophy. NUnit tends to be more flexible and allows for a range of testing styles and can be used for integration tests. It is however flexible enough to allow you to shoot yourself in the foot and allows you to write tests that purists would frown upon. NUnit leaves the choice up to you.
xUnit however tends to be a bit more opinionated and pushes you towards the 'pit of success' with unit testing. In my experience, it is great for green field projects, but it can make testing some poorly architected brown-field projects difficult without refactoring.
In the end, you can't go wrong with any of these test frameworks.
I would like to add one negative aspect of XUnit.
It does not have even a basic documentation, nothing beyond "Get Started". No API, nothing.
Here is a brief Analysis of the 3 Frameworks and ranked by color.
Legend of the Coloring:
MSTest it is good only because it is simple and integrated into Visual Studio and it comes from Microsoft. Its not flexible enough and not very extensible. But more than just a unit test framework, it also supports Integration tests, Web (load) performance tests. The other 2 are pure Unit Test Frameworks. So I would say MSTest is "Jack of all Trades, Master of none."
NUnit is my choice because its been the best consistently long enough. It is more flexible and simple with very good user base, documentation and community support.
XUnit is least of my choice because it follows a slightly different style and personally I do not see much advantage over NUnit. In some cases you will need to do some refactoring your code to fit into the test cases. But this is just my opinion as there are many who prefers XUnit.
I have listed down the comparisons in two categories:
Framework:
Implementation:
The source of these comparisons
I am working on a project in where I decided to use unit tests. This was new to me but after researching I feel pretty confident I am doing it correctly creating mock objects and testing that the correct methods are called. This is working great but now I would like to actually run some tests that use the actual database and external components. How should I go about testing the actual execution of code? I do not want these tests to run when I run all tests. Is there a way to accomplish this using built in testing in vs2012?
Not that I'm aware of.
What you could do is create a separate project for your integration tests and then in the test explorer, separate your tests by class. Or some other logical seperation.
the current implementation that exists seems to specifically have unit testing mainly in mind, but that should change with 2013 and a stronger push towards TDD and Agile development.
Have a look here :http://msdn.microsoft.com/en-us/library/ms243147(v=vs.80).aspx
Scroll down to "Attributes for Identifying and Sorting Tests". There might be something there that's useful that you can use.
Roy Osherove, author of The Art Of Unit Testing, has commented on a blog that of the many things NUnit supposedly does better, it being much faster is one of them.
My question is how much faster though, if at all? Are we talking an order of magnitude? 10%? 50%?
I'm asking this because for the moment I can't compare the two. I am trying to setup my test project to be in a dual-mode so that I can switch between them. Unfortunately, I am having a problem with NUnit integrating with the latest version of Microsoft Moles, and also NUnit is conflicting with a third party library (appears log4net related).
So far MSTest seems so much easier to use within Visual Studio 2008. All of the version issues and compatibility problems with NUnit (at least for me) is steering me towards choosing MSTest as the framework for the project (though I may keep the dual mode option). Another plus for MSTest is that I can still use most of NUnit asserts with:
using Assert = NUnit.Framework.Assert;
using Is = NUnit.Framework.Is;
But.... If speed is truly much faster in NUnit, then I'd prefer to use it, despite the pain points.
Lastly, has there been any speed improvement in VS2010 for MSTest?
Well I took the time to remove Microsoft Moles and the production code that dealt with log4net related issues so that I could actually compare the two. Then I ran tests in both MSTest 2008 and NUnit 2.5.2.
What I found out is that MSTest better reports the duration time of each individual test. If the test is fast enough in NUnit it gets logged as being 0 seconds both in the TestResults.xml file and in the GUI when one clicks on the properties of a test. Nonetheless, I tried to compare the sum of all test speeds and in some cases NUnit is faster and in other cases MSTest is faster. When one is faster than the other it's by about 30%.
Now, where NUnit definitely seems faster is the wait time before the unit tests run. When I attach the GUI (or console) to the VS Debugger, and run the test project, it takes about 3-6 seconds for NUnit to launch and load before the tests can execute. With MSTest it takes between 15-20 seconds. For MSTest it doesn't matter if there is just 1 test or 26, this load time seems to be the same. As regards to how these wait times scale as the test project gets bigger, say in the thousands of unit tests, I can't comment on the difference, though I'd be very interested in knowing.
There also seems to be a little bit of delay in MSTest while the tests are running, perhaps to update the results pane. Thus, I suspect when people say that NUnit is much faster than MSTest, it is because of the loading and updating delays, but the actual test execution time appears to be very similar.
I have used both MSTest and NUnit (the first more than the latter), and I can't say I noticed any big differences between the two of them regarding speed (don't get me wrong, the difference might be there, I just haven't noticed it).
The reason I choose MSTest is its integration with Visual Studio, as it makes thigns a lot simpler. Additionally, once I had some issues running a couple of tests because the NUnit tests ran in a different thread appartment than MSTests.
With the release of BizTalk 2009 we can now use Visual Studio Team Edition 2008 to leverage some of the team test features.
We can unit test maps, schemas, and pipelines ... but from what I can tell, that's about it.
How would you unit test orchestrations for instance?
Well I have been using BizUnit for years already and plan to continue to do so;
What I like about BizUnit is that it is extensible and opensource
So ... Who can give me a few reasons why I should be ditching BizUnit in favor of Team Test in Visual Studio?
You can unit test your orchestration using Bizmock http://www.codeplex.com/bizmock I just looked at it briefly, didn't get a chance to use it on real time projects. But according to the documentation and sample its looks like perfect tool to unit test your orchestration.
I looked at the BizUnit page, and it seems to me that you can use BizUnit in conjunction with Visual Studio Team Test. BizUnit is a declarative testing framework that can augment VS Team Test's capabilities. BizUnit does not require VS Team Test, but VS Team Test can provide an excellent mechanism for driving BizUnit test cases.
The new features of BizTalk 2009 will give you much better support for debugging maps, testing schemas and pipelines but as far as orchestrations go, you still need something like BizUnit to cover that need. I would say the build components alone are a reason to upgrade, so you can setup continuous builds within TFS and the like, which before you had to use an open source solution. So specifically to your question you wouldn't ditch BizUnit for anything and its still needed.
-Bryan
In my experience the answer to how you test BizTalk orchestations depends on the type of application that you are writing in BizTalk.
For example BizUnit works very well for testing the usual integration scenarios, but less so (in my opinion) where the BizTalk solution to be tested is written more like a conventional [middle-tier] application.
In the solution I am developing I have several nested orchestrations invoked from a "parent" orchestration with call-orchestration shape. These child orchestrations have logic/work-flow that can and still need to be tested in isolation.
I have been looking at exercising these from a VS Unit test via a "wrapper" orchestration which accepts the required parameters and invokes the orchestration to be tested (via start or call orchestration shape). The wrapper itself is invoked via an incoming message originating at the unit test.
I am also interested in evaluating bizmock.
I've recently started work on the Compact Framework and I was wondering if anyone had some recommendations for unit testing beyond what's in VS 2008. MSTest is ok, but debugging the tests is a nightmare and the test runner is so slow.
I see that NUnitLite on codeplex is an option, but it doesn't look very active; it's also in the roadmap for NUnit 3.0, but who knows when that will come out. Has anyone had any success with it?
What we've done that really improves our efficiency and quality is to multi target our mobile application. That is to say with a very little bit of creativity and a few conditional compile tags and custom project configurations it is possible to build a version of your mobile application that also runs on the desktop.
If you put all your business logic you need tested in a separate project/assembly then this layer can be very effectively tested using any of the desktop tools you are already familiar with.
We use NUnitLite, although I think we did have had to add some code to it in order for it to work.
One of the problems we found is that if you are using parts of the platform that only exist in CF, then you can only run those tests in NUnitLite on an emulator or Windows Mobile device, which makes it hard to run the tests as part of an integrated build process. We got round this by added a new test attribute allowing you to disable the tests what would only run on the CF (typically these would be p/invoking out to some windows mobile only dll).