I'm developing a project in Visual Studio 2012 with some other developers. In my project, every one is connected together with TFS 2012.
I'm writing a lot of unit-tests that testing the classes on my project.
In a perfect world, I wish to:
Run all my unit tests every night. It's will be nice if I could control the scheduling (e.g. on Sunday run X tests and on Monday run Y tests).
Getting report every day of which tests failed/passed. It would be nice to get an email sent to me.
Browse running history: view the running history of specific test.
Is that possible?
I've already viewed the solution "Run tests after build" and found it ineffective. It's bothersome that after each [build -> build solution] it's starts to run 100+ unit-tests.
I would suggest you use something like TeamCity, this is Continuous integration Services, this will be build run test every time you check in,
Jenkins is a continuous build machine, which it sounds like would meet your needs. You may have to intall some plugins, but it can do what you are talking abotu
Related
We've got several thousand unit tests that run in one of our projects. These currently take over ten minutes to run, and I'm trying to reduce this time.
One of the things I'm looking at is determining which tests are impacted by the latest code changes and only run those impacted tests. This seems to be a feature that TFS has, but all of the tutorials are from TFS2010 and the old XAML builds and not for the new vNext builds in TFS2015.
Does anyone have any experience in getting this to work?
Following this tutorial:
https://msdn.microsoft.com/en-us/library/ff576128(v=vs.100).aspx
I've defined a .testsettings file, and referenced it from a .runsettings file, and then in TFS2015 build I've set the .runsettings file as the Run Settings here:
But there doesn't seem to be any test impact files generated.
Answer from Microsoft:
Hi Steve,
Thanks for your post.
I tested it in my TFS 2015 vNext build and received the same result.
It seems there’s no the Analysis test impact = True/False option in
TFS 2015 vNext build Test task, this option is exist in XAML build
definition.
For this scenario, please submit it to User Voice site at:
http://visualstudio.uservoice.com/forums/121579-visual-studio,
Microsoft engineers will evaluate it seriously.
https://social.msdn.microsoft.com/Forums/en-US/610e304d-608e-49e9-8fcc-e65617828a6f/determine-impacted-tests-during-vnext-build?forum=tfsbuild
Another day, another problem with VS2012 and unit tests: Our solution has three test projects, together containing 335 different unit tests. All of these tests successfully run locally on different machines, but when checked in to our TFS 2012 build server, each time only 235 tests are being discovered.
There are 15 different test classes containing the unit tests, some of them using the new Fakes framework, others MOQ, some none or both. All of the classes reside in test projects (assemblies) ending in Test, the classes themselves are all ending in Test.
Some classes are not being discovered at all during the test run as part of the build process, in other classes e.g. 8 out of 12 tests are discovered and run.
Taking the "8 out of 12" example, the [ProjectNameTest].dll that's being generated shows all 12 tests using .NET Reflector 7.0, it's just so that some of the tests aren't run. I also cannot find any similarities (e.g. tests using Fakes / MOQ / ...) among the tests not run / discovered.
The build settings being used define the Test Sources Spec as **\*.dll (it used to be **\*test*.dll, but I changed this just to make sure that all files are taken into account).
For the Run Settings I initially started of with no Run Settings File and CodeCoverageEnabled. As part of trying to solve the problems I also changed this to a user defined UnitTest.runsettings file, unfortunately with the same result.
Does anybody know why there are all these problems? What might be the reasons? VS2010 and Moles worked a lot better, VS2012 has been causing a lot of trouble recently.
Update
Months later we finally updated the TFS installation to Update 3, but this made things even worse. Unit tests on the build server don't run anymore at all. Enabling dumps and analysing them showed that a StackOverflowException is thrown, but there is no indicator about the test that might be causing it. Again, locally on different developer systems the tests run without any problems.
We are considering opening a call at Microsoft as we've spent far too much time getting the whole thing to work (again). Moles were a whole lot better... I will keep this updated as soon as there is some kind of progress.
Update 2
Another few weeks have passed, and no real solution yet. As we have moved from VS2012 to VS2013 (Premium each time) now, we discovered that all tests are now run as part of the build process (again). Unfortunately, we still have problems getting all tests to pass - locally, everything runs fine on multiple systems, on the build server about ten percent of test fail. Investigation to be continued...
Usually, when tests pass locally but not on the build server, you probably need to include your assembly dependencies or if your tests are using databases you are connecting to the database properly.
Finally an answer was found - the problems with test failing randomly on the TFS build server were caused by using the Microsoft Fakes Framwork. On our local systems, everything worked as expected - we had Fakes defined in our different unit test projects (generally one test project for each "normal" project).
On the build server all the fakes were generated in the same directory, therefore depending on the build order of the projects (which in parts may be different for each build), the fakes generated first were overwritten by the fakes from the following project etc.
Instead of having the fake definitions as part of every project in our solution, we changed it so that we have a solution wide definition of fakes (e.g. System.Web.Fakes.dll) that we link to from each test project where we need it (Add existing item => Add as link). This solved the problems with running the unit tests on the build server, all tests pass again.
In our solution we have several test projects. Most of them contain unit tests that run fast but some are for integration, acceptance and UI tests which take quite some time to finish.
Now I'd like to have a convenient way of running only the unit tests. Either by using the hotkey to run all or using the new feature to run tests after each build.
Unfortunately VS also runs all the slow tests as well. In VS10 I grouped the tests in Test View by project and selected only the ones containing unit tests. It's not as easy as I wished it to be but at least it was possible. Now there doesn't seem to be a way around running all tests in a solution.
The only alternative that I see at the moment is to remove all but the unit tests from the solution but this would limit the refactoring capabilities.
So does anyone know how to easily group and run only unit tests in the new VS?
In Visual Studio 2012 Update 1 CTP, Test Explorer has added the option to group tests by Project or Traits (Categories).
Please go to this link to see what's new there.
You can use the search filter in Unit Text Explorer.
In the new Test Exlorer shipped with 2012, you can group tests by 'Test Results', 'FileName',... etc but in RTM build group/search by 'TestCategory' is not present from Test Explorer. However there is a filter in commandline and TeamBuild based on TestCategory (Running selective unit tests in VS 2012 RC using TestCaseFilter) in VS2012.
From a member of VS dev Team, they understood the importance of this missing feature. It is in the backlog, and it should be available in furutre builds/updates.
The following blog provides more details: http://blogs.msdn.com/b/vikramagrawal/archive/2012/07/23/running-selective-unit-tests-in-vs-2012-rc-using-testcasefilter.aspx
As of Visual Studio 2012 Update 2 (currently on CTP) now we are able to run a subset of unit test using "Playlists" or based on "Lists & Classes". More information here: http://blogs.msdn.com/b/visualstudioalm/archive/2013/01/30/first-ctp-for-visual-studio-update-2.aspx#playlists
Original Answer:
AFAIK, The new unit testing framework in VS2012 "supports selective execution based on filtering condition through TestCaseFilter". Unfortunately, currently there is not support to do this through the user interface but it is in the product group's backlog. You can use this approach through Test Build or command line. Please refer to the following blog: http://blogs.msdn.com/b/vikramagrawal/archive/2012/07/23/running-selective-unit-tests-in-vs-2012-rc-using-testcasefilter.aspx
EDIT: Following the MSDN documentation to use test categories to group tests.
http://msdn.microsoft.com/en-us/library/dd286595.aspx
Also, refer to this question: How to exclude certain tests in the Visual Studio 2012 Test Runner?
It shows an example of filtering based on the test path.
I'd recommend using TestDriven.Net. Group all Unit Testing projects in one solution folder, and all other tests in another. Then just right click the "Unit Tests" folder and choose "Run All Tests". I always favor this great test runner of the VS native testing tools.
Another approach is using NCrunch - It runs all you tests in the background while you code. It handles integration tests pretty well, as they run in parallel, in the background so it doesn't block you (you also choose which test projects are run). One catch though - I believe it doesn't support VS2012 yet, but probably soon.
I was also missing this feature... (We did the same as you did: grouping by project or sometimes by namespace). But I found that I could somehow workaround this using the test explorer search capabilities:
https://msmvps.com/blogs/deborahk/archive/2012/09/16/searching-and-grouping-in-vs-2012-test-explorer.aspx
In my case I could identify the specific tests I needed to run using the search filter on FullName (use dropdown on search box, or type FullName:"UnitTest")
Its not the same, but it works for me
MS Test is killing me.
It is so slow compared to NUnit. I am stuck with it because I need to be able to get Test Results into TFS easy. (Plus it works better with Pex and I am using that too).
But I would really like it to go faster. Even just a bit faster would be nice.
Has anyone made a test runner for MS Test tests that goes faster than the one embedded in Visual Studio?
The Gallio platform provides a test adapter for MSTest. You can then run your tests with any runner supporting Gallio, including built-in runners such as Echo or Icarus, and 3rd party runners such as TD.Net or R#.
The non-GUI test runners are usually slightly faster. Thus you may want to try Echo or the PoSh snap-in.
You should be able to integrate NUnit into TFS with some custom build tasks (which some people have done). If you really wanted to get crazy you could transform NUnit test results into .TRX files and import them directly using the MSTEST command line options
TestDriven.Net is probably the fastest one.
Then goes msTest runner embedded into Visual Studio.
I found that ReSharper test runner is slower than built-in, especially if the solution is big.
At the time I was researching, ReSharper seemed to do a full rebuild of the solution, where msTest was just building changed projects.
So your best choice would be TestDriven.Net
Albacore has an MSTest task.
https://github.com/derickbailey/Albacore/wiki/MSTest-Task
You could write a simple rake file that built your project and then ran the MSTest task.
My recommendation is that, initially, you write a project to only run the albacore MSTest task and time how long that takes.
PS: I wouldn't use Gallio because, in my experience, that test runner takes 6 seconds longer than TestDriven.net to run NUnit tests.
PS 2: My real recommendation is to ditch MSTest for xUnit.net or NUnit
From reasons I won't get into, all our unit tests are using the VSTS test framework. I now want to create an MSBuild script that runs the tests, but I don't want to use mstest.exe from various reasons (it's slower, requires Visual Studio installation everywhere, I need to maintain testrunconfig, etc.)
I've seen that TestDriven.net and TeamCity are able to run VSTS tests 'NUnit style', without using mstest.exe. Are you aware of any standalone command line utility that does this?
You can execute Team System Tests (MSTest) in NUnit if you use a special NUnit Addin that recognizes the MS Test Attributes (TestClass, etc).
Exact Magic Software has an open-source "test-adapter" that can do this.
UPDATE: I've reworked Exact Magic's Msts NUnit Adapter for NUnit 2.5.2.
Download here: http://snippetware.googlecode.com/files/ExactMagic.MstsAdapter.zip
Read more about it here: http://www.bryancook.net/2009/10/mstest-nunit-adapter.html
It seems like TeamCity is simply leveraging Gallio to run VS tests. Gallio appears to have msbuild integration and sounds perfect but after a closer look it seems that it would require a VS install just like MSTest as it appears to depend on MS exes:
The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or
${process:VSTESTHOST.EXE} or
${process:QTAGENT.EXE} or
${process:QTAGENT32.EXE} or
${process:QTDCAGENT.EXE} or
${process:QTDCAGENT32.EXE}'.
Host process exited with code: 0
That being said it sounds like at least one person has got it working:
Christoph De Baene - Running MSTest without Visual Studio
It is possible to run MSTests without installing Visual Studio. See how-do-i-use-mstest-without-visual-studio.
I did this so that I could run my tests as part of my CI process. (I am using CC.NET for my CI solution).
I am in a similar situation as you, in that I want to use TestDriven.NET to get code coverage stats. But, I am running into problems. My first problem is that I am using AssemblyInitialize attributes to initialize a database connection. This isn't supported by NUnit so about half of my tests fail whereas they run fine under MSTest.
So, it seems that translating tests from one test framework to another has pitfalls. If you are aware of that, then go forth, but it might be better to try and keep consistent on one test framework.
We run VSTS tests using msbuild TestToolsTask on a Cruise Control server. This does not use the MSTEST executable -- the condition you ask for -- but does use a variety of TFS dependencies.
Note that we are migrating tests off of the VSTS test framework for NUnit, mostly because we can create extensions for NUnit to perform useful tasks.