From reasons I won't get into, all our unit tests are using the VSTS test framework. I now want to create an MSBuild script that runs the tests, but I don't want to use mstest.exe from various reasons (it's slower, requires Visual Studio installation everywhere, I need to maintain testrunconfig, etc.)
I've seen that TestDriven.net and TeamCity are able to run VSTS tests 'NUnit style', without using mstest.exe. Are you aware of any standalone command line utility that does this?
You can execute Team System Tests (MSTest) in NUnit if you use a special NUnit Addin that recognizes the MS Test Attributes (TestClass, etc).
Exact Magic Software has an open-source "test-adapter" that can do this.
UPDATE: I've reworked Exact Magic's Msts NUnit Adapter for NUnit 2.5.2.
Download here: http://snippetware.googlecode.com/files/ExactMagic.MstsAdapter.zip
Read more about it here: http://www.bryancook.net/2009/10/mstest-nunit-adapter.html
It seems like TeamCity is simply leveraging Gallio to run VS tests. Gallio appears to have msbuild integration and sounds perfect but after a closer look it seems that it would require a VS install just like MSTest as it appears to depend on MS exes:
The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or
${process:VSTESTHOST.EXE} or
${process:QTAGENT.EXE} or
${process:QTAGENT32.EXE} or
${process:QTDCAGENT.EXE} or
${process:QTDCAGENT32.EXE}'.
Host process exited with code: 0
That being said it sounds like at least one person has got it working:
Christoph De Baene - Running MSTest without Visual Studio
It is possible to run MSTests without installing Visual Studio. See how-do-i-use-mstest-without-visual-studio.
I did this so that I could run my tests as part of my CI process. (I am using CC.NET for my CI solution).
I am in a similar situation as you, in that I want to use TestDriven.NET to get code coverage stats. But, I am running into problems. My first problem is that I am using AssemblyInitialize attributes to initialize a database connection. This isn't supported by NUnit so about half of my tests fail whereas they run fine under MSTest.
So, it seems that translating tests from one test framework to another has pitfalls. If you are aware of that, then go forth, but it might be better to try and keep consistent on one test framework.
We run VSTS tests using msbuild TestToolsTask on a Cruise Control server. This does not use the MSTEST executable -- the condition you ask for -- but does use a variety of TFS dependencies.
Note that we are migrating tests off of the VSTS test framework for NUnit, mostly because we can create extensions for NUnit to perform useful tasks.
Related
I am just starting to work with QuantLib project. There are quite a few BOOST unit tests defined, hence the question: is there an easy way to run these tests selectively from Visual Studio (and see results)?
I've only used unit testing in .Net languages so far, and there you can either use the built-in test runner if you use MsTest, or some add-on (I used Resharper and TestDriven.NET) for other frameworks. There you can choose which tests you run, and you can run them with the debugger attached (which is unfortunately useful sometimes). Are there similar add-ons for BOOST tests in unmanaged C++?
Yes you can!
If your boost unit test project is defined correctly, then you can run the entire test project from VS10 debugger by clicking
right click on project --> Debug --> Start new instance
This will run the tests inside VS framework which will give you all the debugger goodies.
For running specific test suites/cases, just edit the project command line arguments under
right click on project --> Properties --> General --> Debugging
and add the parameters according to the links in the other answers
you can run any test case or test suite via command line. running tests by name
it even allows wildcards.
ReSharper C++ supports discovery and selective running/debugging of Boost tests (with Boost 1.59 and later).
I'm using the Boost Unit Test Adapter. It's free and has a very nice output. It can also list the time the single tests consumed and you can start single tests and group of tests.
You can download it here:
https://visualstudiogallery.msdn.microsoft.com/5f4ae1bd-b769-410e-8238-fb30beda987f?SRC=VSIDE
In our solution we have several test projects. Most of them contain unit tests that run fast but some are for integration, acceptance and UI tests which take quite some time to finish.
Now I'd like to have a convenient way of running only the unit tests. Either by using the hotkey to run all or using the new feature to run tests after each build.
Unfortunately VS also runs all the slow tests as well. In VS10 I grouped the tests in Test View by project and selected only the ones containing unit tests. It's not as easy as I wished it to be but at least it was possible. Now there doesn't seem to be a way around running all tests in a solution.
The only alternative that I see at the moment is to remove all but the unit tests from the solution but this would limit the refactoring capabilities.
So does anyone know how to easily group and run only unit tests in the new VS?
In Visual Studio 2012 Update 1 CTP, Test Explorer has added the option to group tests by Project or Traits (Categories).
Please go to this link to see what's new there.
You can use the search filter in Unit Text Explorer.
In the new Test Exlorer shipped with 2012, you can group tests by 'Test Results', 'FileName',... etc but in RTM build group/search by 'TestCategory' is not present from Test Explorer. However there is a filter in commandline and TeamBuild based on TestCategory (Running selective unit tests in VS 2012 RC using TestCaseFilter) in VS2012.
From a member of VS dev Team, they understood the importance of this missing feature. It is in the backlog, and it should be available in furutre builds/updates.
The following blog provides more details: http://blogs.msdn.com/b/vikramagrawal/archive/2012/07/23/running-selective-unit-tests-in-vs-2012-rc-using-testcasefilter.aspx
As of Visual Studio 2012 Update 2 (currently on CTP) now we are able to run a subset of unit test using "Playlists" or based on "Lists & Classes". More information here: http://blogs.msdn.com/b/visualstudioalm/archive/2013/01/30/first-ctp-for-visual-studio-update-2.aspx#playlists
Original Answer:
AFAIK, The new unit testing framework in VS2012 "supports selective execution based on filtering condition through TestCaseFilter". Unfortunately, currently there is not support to do this through the user interface but it is in the product group's backlog. You can use this approach through Test Build or command line. Please refer to the following blog: http://blogs.msdn.com/b/vikramagrawal/archive/2012/07/23/running-selective-unit-tests-in-vs-2012-rc-using-testcasefilter.aspx
EDIT: Following the MSDN documentation to use test categories to group tests.
http://msdn.microsoft.com/en-us/library/dd286595.aspx
Also, refer to this question: How to exclude certain tests in the Visual Studio 2012 Test Runner?
It shows an example of filtering based on the test path.
I'd recommend using TestDriven.Net. Group all Unit Testing projects in one solution folder, and all other tests in another. Then just right click the "Unit Tests" folder and choose "Run All Tests". I always favor this great test runner of the VS native testing tools.
Another approach is using NCrunch - It runs all you tests in the background while you code. It handles integration tests pretty well, as they run in parallel, in the background so it doesn't block you (you also choose which test projects are run). One catch though - I believe it doesn't support VS2012 yet, but probably soon.
I was also missing this feature... (We did the same as you did: grouping by project or sometimes by namespace). But I found that I could somehow workaround this using the test explorer search capabilities:
https://msmvps.com/blogs/deborahk/archive/2012/09/16/searching-and-grouping-in-vs-2012-test-explorer.aspx
In my case I could identify the specific tests I needed to run using the search filter on FullName (use dropdown on search box, or type FullName:"UnitTest")
Its not the same, but it works for me
I have 80+ VS2010 solutions, each contains some unit test projects.
All solutions are merged into one big solution before build process.
All tests (or some test subset) are executed after the successful build.
I want to enable test code coverage for all tests to get exact code coverage for all output assemblies.
My question is: how to enable code coverage without using Visual Studio?
Note: I'm using TFS2010 (MSBuild) to build merged solution. VS 2010 Premium is installed on the build server. MSTest is used for test execution.
You can use JetBrain's TeamCity Professional. It is a CI server that supports executing unit tests and calculating code coverage. It is free for small installations.
I think you need to consider deploying a code coverage tool, see here for a comparison (provided you implement .net).We use NCover, which integrated in our TFS-Build in it's console variant and, although it's not trivial to set it up, we 're very satisfied with it.In this post I had briefly described how we inserted NCoverin our build, this might we useful to you even if you go with another tool.
If you create a Vsmdi file in your large solution (ms test will usually do this for you) you can use this to tell the build which assemblies you want to instrument.
This will only provide code coverage for assemblies that have tests run against them. If you're using testrun.config files to decide which tests you want to run, this should be all you need. The code coverage results should then be published to the build drop location
Edit:
This blog post looks like it covers setting up code coverage
MS Test is killing me.
It is so slow compared to NUnit. I am stuck with it because I need to be able to get Test Results into TFS easy. (Plus it works better with Pex and I am using that too).
But I would really like it to go faster. Even just a bit faster would be nice.
Has anyone made a test runner for MS Test tests that goes faster than the one embedded in Visual Studio?
The Gallio platform provides a test adapter for MSTest. You can then run your tests with any runner supporting Gallio, including built-in runners such as Echo or Icarus, and 3rd party runners such as TD.Net or R#.
The non-GUI test runners are usually slightly faster. Thus you may want to try Echo or the PoSh snap-in.
You should be able to integrate NUnit into TFS with some custom build tasks (which some people have done). If you really wanted to get crazy you could transform NUnit test results into .TRX files and import them directly using the MSTEST command line options
TestDriven.Net is probably the fastest one.
Then goes msTest runner embedded into Visual Studio.
I found that ReSharper test runner is slower than built-in, especially if the solution is big.
At the time I was researching, ReSharper seemed to do a full rebuild of the solution, where msTest was just building changed projects.
So your best choice would be TestDriven.Net
Albacore has an MSTest task.
https://github.com/derickbailey/Albacore/wiki/MSTest-Task
You could write a simple rake file that built your project and then ran the MSTest task.
My recommendation is that, initially, you write a project to only run the albacore MSTest task and time how long that takes.
PS: I wouldn't use Gallio because, in my experience, that test runner takes 6 seconds longer than TestDriven.net to run NUnit tests.
PS 2: My real recommendation is to ditch MSTest for xUnit.net or NUnit
I'm trying to work out how to run xUnit tests from TFS 2010.
I found some articles on how to achieve that with the older version (http://jonnekats.wordpress.com/2009/05/07/integrate-xunit-tests-into-your-daily-team-build/, http://weblogs.asp.net/mehfuzh/archive/2009/08/25/configuring-team-build-using-xunit.aspx).
These approaches don't work anymore with TFS 2010 because builds are no longer MSBuild files but instead are Workflows with different tasks.
What I would like to achieve is something similar to what I've got on my dev machine:
- build everything
- run the .xunit project file
- check the results
All that I can see on the web are custom build steps which I can't/won't use because I'll have to configure them for each single unit test assembly and they'll probably get completely messed up in TFS.
Any input would be appreciated.
Why dont use "InvokeProcess" in TFS Build workflow and invoke xUnit command line interface with the necessary xUnit project files as parameters?
You can capture the output of the command line by setting InvokeProcess->stdout to write a build message.
Take a look at the xUnit work item Support VSTS Test Runner on codeplex. You can find there a VS2010 extension in Beta version.