The company I'm working at uses xUnit to write Integration tests. xUnit works perfectly for us but we will like to extract more statistical information out of runs. For example - "How many times did this specific Test-Case has been failing in the last month", and maybe even sort it out nicely on a pie chart.
Since we have a Microsoft Test Manager license, I did some research on it, and it seems like it does support more detailed reports. I also like the coupling between Manual Test-Cases to Automation Test-Cases, and the fact you easily identify how much of your Test-Cases are automated.
Sadly enough, Test Manager only support MSTest integration out of the box. I did noticed however that the MSTest.ext alternative - VSTest.exe is able to run xUnit tests, and even output TRX result file. Is there any way to integrate xUnit (or nUnit) to the Test Manager somehow? Has anyone done so in the past? we prefer to use Test Manager, but I'm interesting to know if there is an alternative that support a couple Test-Cases with Automated Test-Cases and a way to get statistical information about multiple-runs.
Thank you.
This tool will allow you to associate NUnit and xUnit test cases with Microsoft Test Manager.
https://github.com/JakeGinnivan/TestCaseAutomationAssigner
After some research on the Microsoft Forum (and personally speaking with Microsoft representative), It seems like it's not possible to use xUnit with MS Test Manager.
We decided not to use MS Test Manager and handle all our test runs using VSTest.exe and xUnit categories.
Edit: It's now possible to use an external tool for that called "TestCaseAutomationAssigner". See Jeff's answer for more information.
Related
Currently, I am using jacoco report tool to know how much coverage for my system. But this system is quite old and has many existing Fitnesse tests. I am newbie on this system and want to learn it by reading its Fitnesse test. My problem is I dont know what Fitnesse test is for what specific class.Beside that When I see a class is covered and I dont know which Fitnesse test covered this class. My system has more than 500 Fitnesse Tests.
JaCoCo does not provide this out of the box.
There are some third-party JaCoCo integrations such as SonarQube that allow to obtain information about relation between JUnit or TestNG test to code. Don't know much about Fitnesse, so can't tell whether this will work with it or not.
However referring to the same link: general principle to obtain such information using JaCoCo (and that's how SonarQube does) - is to measure coverage separately for each test and save data into place dedicated to this test.
Also IDEs are able to show coverage (there are EclEmma plugin for Eclipse based on JaCoCo and other tools for other IDEs), so if you're able to run Fitneese test in IDE, then this principle can be applied even manually by running one test after another within IDE.
Also have a look at another code coverage tool for Java that is commercial and named Clover.
I realize there have been a LOT of questions on this topic but somehow I haven't found one that addressed my needs.
My team is looking to start automated Unit testing our application. We have never done it before and nobody on the team has much experience with it. I have been asked to research and find a framework for automating our Unit Testing. So far, I have narrowed the choices down to MSTest, NUnit and xUnit.
All across the internet, I read negative reviews about MSTest but it seems the most convenient tool to me for our purpose.
1) Our application is .NET
2) We use licensed VS IDE for our development across the team.
3) Source Control is Team Foundation Server 2010
4) We plan to integrate CI/CD as part of our software delivery process.
I feel that MSTest integrates right into this setup and provides the cleanest interface for us to work on. Is there a significant advantage with NUnit or xUnit over MSTest that we should consider?
I am biased because I work on NUnit, but the advantage of NUnit or xUnit is that both frameworks offer more functionality like data driven tests, parallel execution and a host of advanced features.
That said, there is nothing wrong with MSTest especially if your team is not very experienced with unit testing. It is well integrated with Visual Studio and with TFS, so adding tests is fairly painless. Both NUnit and xUnit integrate well too, but they may require more setup.
Go ahead and start with MSTest. It is an easy introduction to unit testing. If you start running into some of the limitations of MSTest, then it is a fairly easy migration to NUnit or xUnit. For simple cases, it is just a matter of changing your NuGet packages and then find/replace the attributes.
When it comes to choosing between xUnit and NUnit, both are great choices but each has a different philosophy. NUnit tends to be more flexible and allows for a range of testing styles and can be used for integration tests. It is however flexible enough to allow you to shoot yourself in the foot and allows you to write tests that purists would frown upon. NUnit leaves the choice up to you.
xUnit however tends to be a bit more opinionated and pushes you towards the 'pit of success' with unit testing. In my experience, it is great for green field projects, but it can make testing some poorly architected brown-field projects difficult without refactoring.
In the end, you can't go wrong with any of these test frameworks.
I would like to add one negative aspect of XUnit.
It does not have even a basic documentation, nothing beyond "Get Started". No API, nothing.
Here is a brief Analysis of the 3 Frameworks and ranked by color.
Legend of the Coloring:
MSTest it is good only because it is simple and integrated into Visual Studio and it comes from Microsoft. Its not flexible enough and not very extensible. But more than just a unit test framework, it also supports Integration tests, Web (load) performance tests. The other 2 are pure Unit Test Frameworks. So I would say MSTest is "Jack of all Trades, Master of none."
NUnit is my choice because its been the best consistently long enough. It is more flexible and simple with very good user base, documentation and community support.
XUnit is least of my choice because it follows a slightly different style and personally I do not see much advantage over NUnit. In some cases you will need to do some refactoring your code to fit into the test cases. But this is just my opinion as there are many who prefers XUnit.
I have listed down the comparisons in two categories:
Framework:
Implementation:
The source of these comparisons
i have project in .net , i want to test it.
But i dont know anything about testing and its method.
how can i go ahead with testing.
which method is better for me for begining?
Is there anything to decide which testing method is taken into account for better result?
There is no "right" or "wrong" in testing. Testing is an art and what you should choose and how well it works out for you depends a lot from project to project and your experience.
But as a professional Tester Expert my suggestion is that you have a healthy mix of automated and manual testing.
AUTOMATED TESTING
Unit Testing
Use NUnit to test your classes, functions and interaction between them.
http://www.nunit.org/index.php
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.peterkrantz.com/2005/selenium-for-aspnet/
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
Since it is not clear about the scale of the project you have, all you need to do is make sure:
Your tests are trustworthy - you should know they are telling u the truth.
Repeatable
Consistent - If you repeat test with same test data it should provide same output.
Proves you are covering all the problem areas.
To get this you can use:
Standard way : NUnit, MbUnit (myFav) or xUnit (havent got around to working with it) or MSTest
Quick and Dirty : Console app (not cool, not so flexible)
If you are using .Net, I'd recommend checking out NUnit. It's a great testing framework to use.
As far as learning about the "testing method", there are many different ways to test an application. When using a tool like NUnit, for example, you are writing automated tests which run without user interaction. In these types of tests, you typically write tests for each of the public methods in your application, and you ensure that given known inputs, these methods produce the expected outputs. Over time as the application changes (via enhancements, bug fixes, etc.) you have a core set of tests that you can re-run to ensure nothing breaks as a result of the changes. You can also do failure testing to ensure that given an invalid set of inputs to a method, it throws the proper exceptions, etc.
Besides automated testing with a tool like NUnit, it's also important to ensure that your end users test the product. "End users" here could be a Quality Assurance group in your company, or it could be the actual customer. The point is that you need to ensure that someone actually uses your application to make sure it works as expected, because no matter how good the automated tests are, there will still be many things you won't think of that your users will discover. One way to approach this type of testing is to write test scenarios, and have your users execute them to make sure the scenario results in the correct behavior.
I think the best testing approach combines both of the above, namely automated testing and user testing (with documented test scenarios).
With the release of BizTalk 2009 we can now use Visual Studio Team Edition 2008 to leverage some of the team test features.
We can unit test maps, schemas, and pipelines ... but from what I can tell, that's about it.
How would you unit test orchestrations for instance?
Well I have been using BizUnit for years already and plan to continue to do so;
What I like about BizUnit is that it is extensible and opensource
So ... Who can give me a few reasons why I should be ditching BizUnit in favor of Team Test in Visual Studio?
You can unit test your orchestration using Bizmock http://www.codeplex.com/bizmock I just looked at it briefly, didn't get a chance to use it on real time projects. But according to the documentation and sample its looks like perfect tool to unit test your orchestration.
I looked at the BizUnit page, and it seems to me that you can use BizUnit in conjunction with Visual Studio Team Test. BizUnit is a declarative testing framework that can augment VS Team Test's capabilities. BizUnit does not require VS Team Test, but VS Team Test can provide an excellent mechanism for driving BizUnit test cases.
The new features of BizTalk 2009 will give you much better support for debugging maps, testing schemas and pipelines but as far as orchestrations go, you still need something like BizUnit to cover that need. I would say the build components alone are a reason to upgrade, so you can setup continuous builds within TFS and the like, which before you had to use an open source solution. So specifically to your question you wouldn't ditch BizUnit for anything and its still needed.
-Bryan
In my experience the answer to how you test BizTalk orchestations depends on the type of application that you are writing in BizTalk.
For example BizUnit works very well for testing the usual integration scenarios, but less so (in my opinion) where the BizTalk solution to be tested is written more like a conventional [middle-tier] application.
In the solution I am developing I have several nested orchestrations invoked from a "parent" orchestration with call-orchestration shape. These child orchestrations have logic/work-flow that can and still need to be tested in isolation.
I have been looking at exercising these from a VS Unit test via a "wrapper" orchestration which accepts the required parameters and invokes the orchestration to be tested (via start or call orchestration shape). The wrapper itself is invoked via an incoming message originating at the unit test.
I am also interested in evaluating bizmock.
I am looking for any and all suggestions of the best and effective resources that the StackOverflow community has used to better learn MSBuild with an emphasis on integrating unit tests and later static code analysis tools such as FxCop and StyleCop into the build process.
I have tried to find good clear documentation on adding unit tests into my build but I still am searching - even Google searches have come up empty or with just bits and pieces. Ideally I want to add unit tests, report results, and eventually add code coverage statistics, etc into the build results.
I know it has to be in MSDN somewhere but I seem unable to find anything which explains and teaches well. I am using Visual Studio Team System 2008.
Continuous Integration From Theory to Practice by Carel Lotz. It covers the entire scope of your problem, and then some. Well written, complete, and a full sample are all there.
Hands down best resource. Use it as a tutorial first, then use it as a guide, then use it as a reference.
MSDN and others are good for clarifying (or confusing) the details.
Edit: The guide by Carel Lotz uses MBUnit for unit tests (see his earlier document version for NUnit, though you can replace the MBUnit with NUnit pretty easily if you follow the NUnit help files).
Also, it is written to use Cruise Control.NET to run the MSBuild script in various configurations.
Personally, I run unit tests in a secondary MSBuild script, but have found that wrapping the NUnit calls in MSBuild gives more flexibility than running from CCNet directly.
Here's a book that might help: http://blogs.msdn.com/microsoft_press/archive/2009/01/31/sayed.aspx
I guess I need to ask if you are sure you want to use MSBuild directly? Might want to check out WIX as the MSI producing tool - there is an extensive manual and it is built on top of MSBuild.
As for the automating of your tests with reporting and integrating with NUnit, FxCop, NCover, FitNesse, etc - I think the best (free) tool out there is CruiseControl .Net. It works with all of these tools and more. Can do versioning, automated builds with automated testing, creates the reports for each...
Here is a sample of one of my builds...
http://img84.imageshack.us/img84/3664/cruisecontrolnetsamplezn0.jpg http://img84.imageshack.us/img84/3664/cruisecontrolnetsamplezn0.jpg