When testing a Windows Universal class library (.NET 4.6) in Visual Studio 2015, the time required in the "red->green->refactor" cycle is quite long. My test project is just the standard MSTest "Unit Test App" project. Even in the simplest scenario of a brand new solution, brand new subject and test projects, and a basic int Add(int n1, int n2) method, it's taking about 8-15 seconds. This is the time (after making a small code change) from clicking "run test" till the pass/fail is shown.
On my machine (Win 10 pro) conducting the same experiment but with a WPF-based solution yields about 1-2 seconds.
In the UWP scenario, the actual test time itself is listed as 79 ms. The rest of the time is compiling and deploying the unit test app container.
Is there any way to significantly speed up TDD with UWP?
The reason the TDD cycle takes so long is because all UWP code has to run in an AppContainer, and that requires packaging and deployment which is slow.
The way around this problem is to separate as much of your code as you can into a different project that doesn't need to run in an AppContainer and then test that project instead.
The solution for this is to use a portable class library for your application logic instead of a UWP class library. You'll find the portable class library in the new project dialog:
The default settings should work:
You then need to add a reference from the UWP to the portable library so you can consume it. To test the portable library, use a regular non-UWP unit test project:
Note - to make this work you'll need to change the target framework of the unit test project from 4.5.2 (which is the default) to 4.6.
Any unit test you run from the regular unit test project will run as fast as possible because it doesn't need to run in the AppContainer. I tested this out and the inner loop speed was great. Hope that helps!
Related
I have a Windows 8 Store App with two projects within the solution - one project exclusively for tests. I have added NUnit and NUnit Test Adapter (https://www.nuget.org/packages/NUnitTestAdapter/1.0.0) for this through nuget.
My tests are detected - but I cannot run or debug them. Looking at the test output window, I get the following error:
Could not find test executor with URI 'executor://nunittestexecutor/'. Make sure that the test executor is installed and supports .net runtime version 4.0.30319.34003.
I am not using resharper and I am using Visual Studio Professional 2012 Update 3 on Windows 8.1. References for the test project are as follows:
Based on my comments above, below is a workaround. Unfortunately at this stage, Windows 8.1 store Unit Test project types, using NUnit extension wouldn't work due to the different .NET targets. I tried with different Test Unit Adapters including an NUnitTestAdapterWithFramework.
It seems that the issue you haveing was occurring with standard .NET libraries targeting NUnit test adapter but the above NUnitTestAdapterWithFramework must have fixed those issue. See the Q & A section of the NUnitTestExtension
But unfortunatly it seems that this still of an issue that hasn't been fixed for Win8 Store App type Unit Testing. Pretty sure xUnit.NET also not compatible yet with different .NET target types (i,e WinRT)
So what are the options?
a. For your group, you can change them to use MSTest framework. Outcome - Problem Solved no issues.
b. Workaround "linked project". Outcome - Can't *guarantee** but this should also work.
With option 'b'
In your comment you mentioned.
but I'm still not sure what it does or how to implement a 'linked
project', do you have any more information on this? Also, as this is
for a group university project, I was hoping i wouldn't have to force
too many workarounds
When you think about it, it is not really hard work around. It is simple and I'm sure your group would be able to apply this workaround easily.
Please follow the below steps.
Create a separate class library in your solution (you can target .NET framework 4).
Then add NUnit assemblies and the NUnit test adapter as usual.
Right click on this project and select 'Add' then 'Existing Item'
Select the Win8 Store Unit Test project and locate the Unit Test file you want to add. When you add the file, make sure you select 'Add As a Link' button. Please see below.
Now rebuild the solution, close and re-open the UnitTest explorer and you should be able to run those tests.
*The reason I said can't guaranteed. I haven't really written Unit tests against Win8 App. So if your SUT (System Under Test) require special configuration it might cause issues. But I'm not sure.
Finally creating a link files are not that hard if everything works you can continue to do this until NUnit has the support for Win8 Unit Testing. Or the other option is simply change all your Unit Tests to use MSTest framework if possible.
Another day, another problem with VS2012 and unit tests: Our solution has three test projects, together containing 335 different unit tests. All of these tests successfully run locally on different machines, but when checked in to our TFS 2012 build server, each time only 235 tests are being discovered.
There are 15 different test classes containing the unit tests, some of them using the new Fakes framework, others MOQ, some none or both. All of the classes reside in test projects (assemblies) ending in Test, the classes themselves are all ending in Test.
Some classes are not being discovered at all during the test run as part of the build process, in other classes e.g. 8 out of 12 tests are discovered and run.
Taking the "8 out of 12" example, the [ProjectNameTest].dll that's being generated shows all 12 tests using .NET Reflector 7.0, it's just so that some of the tests aren't run. I also cannot find any similarities (e.g. tests using Fakes / MOQ / ...) among the tests not run / discovered.
The build settings being used define the Test Sources Spec as **\*.dll (it used to be **\*test*.dll, but I changed this just to make sure that all files are taken into account).
For the Run Settings I initially started of with no Run Settings File and CodeCoverageEnabled. As part of trying to solve the problems I also changed this to a user defined UnitTest.runsettings file, unfortunately with the same result.
Does anybody know why there are all these problems? What might be the reasons? VS2010 and Moles worked a lot better, VS2012 has been causing a lot of trouble recently.
Update
Months later we finally updated the TFS installation to Update 3, but this made things even worse. Unit tests on the build server don't run anymore at all. Enabling dumps and analysing them showed that a StackOverflowException is thrown, but there is no indicator about the test that might be causing it. Again, locally on different developer systems the tests run without any problems.
We are considering opening a call at Microsoft as we've spent far too much time getting the whole thing to work (again). Moles were a whole lot better... I will keep this updated as soon as there is some kind of progress.
Update 2
Another few weeks have passed, and no real solution yet. As we have moved from VS2012 to VS2013 (Premium each time) now, we discovered that all tests are now run as part of the build process (again). Unfortunately, we still have problems getting all tests to pass - locally, everything runs fine on multiple systems, on the build server about ten percent of test fail. Investigation to be continued...
Usually, when tests pass locally but not on the build server, you probably need to include your assembly dependencies or if your tests are using databases you are connecting to the database properly.
Finally an answer was found - the problems with test failing randomly on the TFS build server were caused by using the Microsoft Fakes Framwork. On our local systems, everything worked as expected - we had Fakes defined in our different unit test projects (generally one test project for each "normal" project).
On the build server all the fakes were generated in the same directory, therefore depending on the build order of the projects (which in parts may be different for each build), the fakes generated first were overwritten by the fakes from the following project etc.
Instead of having the fake definitions as part of every project in our solution, we changed it so that we have a solution wide definition of fakes (e.g. System.Web.Fakes.dll) that we link to from each test project where we need it (Add existing item => Add as link). This solved the problems with running the unit tests on the build server, all tests pass again.
I am just starting to work with QuantLib project. There are quite a few BOOST unit tests defined, hence the question: is there an easy way to run these tests selectively from Visual Studio (and see results)?
I've only used unit testing in .Net languages so far, and there you can either use the built-in test runner if you use MsTest, or some add-on (I used Resharper and TestDriven.NET) for other frameworks. There you can choose which tests you run, and you can run them with the debugger attached (which is unfortunately useful sometimes). Are there similar add-ons for BOOST tests in unmanaged C++?
Yes you can!
If your boost unit test project is defined correctly, then you can run the entire test project from VS10 debugger by clicking
right click on project --> Debug --> Start new instance
This will run the tests inside VS framework which will give you all the debugger goodies.
For running specific test suites/cases, just edit the project command line arguments under
right click on project --> Properties --> General --> Debugging
and add the parameters according to the links in the other answers
you can run any test case or test suite via command line. running tests by name
it even allows wildcards.
ReSharper C++ supports discovery and selective running/debugging of Boost tests (with Boost 1.59 and later).
I'm using the Boost Unit Test Adapter. It's free and has a very nice output. It can also list the time the single tests consumed and you can start single tests and group of tests.
You can download it here:
https://visualstudiogallery.msdn.microsoft.com/5f4ae1bd-b769-410e-8238-fb30beda987f?SRC=VSIDE
Is there actually a good way to unit test MonoTouch projects using NUnit and the MonoDevelop test runner?
I know there is the official MonoTouch unit tests project type, but running tests within the simulator isn't the way I want to go. For now I want to run tests with the MonoDevelop test runner, later everything should work with Jenkins (CI).
I know the limitations about UI specific code, so everything I want to test has nothing to do with MonoTouch itself, it's all about business logic placed within separete projects.
By adding tests to MonoTouch Library type projects, I am getting System.IO.FileNotFoundException's as described here: http://ben.phegan.name/index.php/2011/02/28/monotouch-and-unit-testing/
By using a separate NUnit test project, I can't reference my system under test, because its project type is of type MonoTouch library project, which, of course, has an incompatible target framework (vMonoTouch).
So, there isn't any real alternative to Touch.Unit, is it?
Is there actually a good way to unit test MonoTouch projects using NUnit
Touch.Unit
and the MonoDevelop test runner?
Not really. MonoTouch projects depends on monotouch.dll which needs to execute under iOS (not OSX). So there's a need for the runner to execute on the simulator or devices.
Now there's a few misconceptions in your question:
later everything should work with Jenkins (CI).
Touch.Unit is already used with continuous builds / integration servers (as long as they are running OSX) using both the iOS simulator and/or devices. Details are available here.
I know the limitations about UI specific code,
Touch.Unit is not about UI testing. In fact it's pretty bad at UI testing (but that's beside the point).
Touch.Unit is a test runner that execute on iOS. That allows you to use MonoTouch / iOS API inside your own tests (it could be UIKit, but it could be StoreKit, GameKit, *Kit, any Foundation class... it's a quite large world).
So, there isn't any real alternative to Touch.Unit, is it?
Yes. If your business logic is well isolated and does not depend on monotouch.dll then you should be able to build it either as:
a non-MonoTouch project (different project, same sources), i.e. linked with the regular framework; or
link to the sources from within your unit test assembly (which links to the regular framework);
That classic nunit test assembly would then be a regular framework project and will be able to run from the default NUnit runner or from within MonoDevelop unit test runner.
In our solution we have several test projects. Most of them contain unit tests that run fast but some are for integration, acceptance and UI tests which take quite some time to finish.
Now I'd like to have a convenient way of running only the unit tests. Either by using the hotkey to run all or using the new feature to run tests after each build.
Unfortunately VS also runs all the slow tests as well. In VS10 I grouped the tests in Test View by project and selected only the ones containing unit tests. It's not as easy as I wished it to be but at least it was possible. Now there doesn't seem to be a way around running all tests in a solution.
The only alternative that I see at the moment is to remove all but the unit tests from the solution but this would limit the refactoring capabilities.
So does anyone know how to easily group and run only unit tests in the new VS?
In Visual Studio 2012 Update 1 CTP, Test Explorer has added the option to group tests by Project or Traits (Categories).
Please go to this link to see what's new there.
You can use the search filter in Unit Text Explorer.
In the new Test Exlorer shipped with 2012, you can group tests by 'Test Results', 'FileName',... etc but in RTM build group/search by 'TestCategory' is not present from Test Explorer. However there is a filter in commandline and TeamBuild based on TestCategory (Running selective unit tests in VS 2012 RC using TestCaseFilter) in VS2012.
From a member of VS dev Team, they understood the importance of this missing feature. It is in the backlog, and it should be available in furutre builds/updates.
The following blog provides more details: http://blogs.msdn.com/b/vikramagrawal/archive/2012/07/23/running-selective-unit-tests-in-vs-2012-rc-using-testcasefilter.aspx
As of Visual Studio 2012 Update 2 (currently on CTP) now we are able to run a subset of unit test using "Playlists" or based on "Lists & Classes". More information here: http://blogs.msdn.com/b/visualstudioalm/archive/2013/01/30/first-ctp-for-visual-studio-update-2.aspx#playlists
Original Answer:
AFAIK, The new unit testing framework in VS2012 "supports selective execution based on filtering condition through TestCaseFilter". Unfortunately, currently there is not support to do this through the user interface but it is in the product group's backlog. You can use this approach through Test Build or command line. Please refer to the following blog: http://blogs.msdn.com/b/vikramagrawal/archive/2012/07/23/running-selective-unit-tests-in-vs-2012-rc-using-testcasefilter.aspx
EDIT: Following the MSDN documentation to use test categories to group tests.
http://msdn.microsoft.com/en-us/library/dd286595.aspx
Also, refer to this question: How to exclude certain tests in the Visual Studio 2012 Test Runner?
It shows an example of filtering based on the test path.
I'd recommend using TestDriven.Net. Group all Unit Testing projects in one solution folder, and all other tests in another. Then just right click the "Unit Tests" folder and choose "Run All Tests". I always favor this great test runner of the VS native testing tools.
Another approach is using NCrunch - It runs all you tests in the background while you code. It handles integration tests pretty well, as they run in parallel, in the background so it doesn't block you (you also choose which test projects are run). One catch though - I believe it doesn't support VS2012 yet, but probably soon.
I was also missing this feature... (We did the same as you did: grouping by project or sometimes by namespace). But I found that I could somehow workaround this using the test explorer search capabilities:
https://msmvps.com/blogs/deborahk/archive/2012/09/16/searching-and-grouping-in-vs-2012-test-explorer.aspx
In my case I could identify the specific tests I needed to run using the search filter on FullName (use dropdown on search box, or type FullName:"UnitTest")
Its not the same, but it works for me