In Visual Studio 11, I'm facing a complex issue.
I'm using the Unit Test Explorer to run unit-tests. However, it shows both my normal unit tests and my ordered tests, which are basically collections of tests to run in a specific order too.
That doesn't work well for me, because I would love to have the setting "Run tests after build" on. However, as you would expect, what happens is that it runs both all my tests individually out of order and then runs my ordered test as shown in the picture below.
I would really love to be able to run only my ordered tests when the build finishes. I am not interested in third-party plugins/extensions to help me do this.
I ended up using a build server, and then configured my build configuration as follows, which will run all tests with the orderedtest file extension:
Related
Another day, another problem with VS2012 and unit tests: Our solution has three test projects, together containing 335 different unit tests. All of these tests successfully run locally on different machines, but when checked in to our TFS 2012 build server, each time only 235 tests are being discovered.
There are 15 different test classes containing the unit tests, some of them using the new Fakes framework, others MOQ, some none or both. All of the classes reside in test projects (assemblies) ending in Test, the classes themselves are all ending in Test.
Some classes are not being discovered at all during the test run as part of the build process, in other classes e.g. 8 out of 12 tests are discovered and run.
Taking the "8 out of 12" example, the [ProjectNameTest].dll that's being generated shows all 12 tests using .NET Reflector 7.0, it's just so that some of the tests aren't run. I also cannot find any similarities (e.g. tests using Fakes / MOQ / ...) among the tests not run / discovered.
The build settings being used define the Test Sources Spec as **\*.dll (it used to be **\*test*.dll, but I changed this just to make sure that all files are taken into account).
For the Run Settings I initially started of with no Run Settings File and CodeCoverageEnabled. As part of trying to solve the problems I also changed this to a user defined UnitTest.runsettings file, unfortunately with the same result.
Does anybody know why there are all these problems? What might be the reasons? VS2010 and Moles worked a lot better, VS2012 has been causing a lot of trouble recently.
Update
Months later we finally updated the TFS installation to Update 3, but this made things even worse. Unit tests on the build server don't run anymore at all. Enabling dumps and analysing them showed that a StackOverflowException is thrown, but there is no indicator about the test that might be causing it. Again, locally on different developer systems the tests run without any problems.
We are considering opening a call at Microsoft as we've spent far too much time getting the whole thing to work (again). Moles were a whole lot better... I will keep this updated as soon as there is some kind of progress.
Update 2
Another few weeks have passed, and no real solution yet. As we have moved from VS2012 to VS2013 (Premium each time) now, we discovered that all tests are now run as part of the build process (again). Unfortunately, we still have problems getting all tests to pass - locally, everything runs fine on multiple systems, on the build server about ten percent of test fail. Investigation to be continued...
Usually, when tests pass locally but not on the build server, you probably need to include your assembly dependencies or if your tests are using databases you are connecting to the database properly.
Finally an answer was found - the problems with test failing randomly on the TFS build server were caused by using the Microsoft Fakes Framwork. On our local systems, everything worked as expected - we had Fakes defined in our different unit test projects (generally one test project for each "normal" project).
On the build server all the fakes were generated in the same directory, therefore depending on the build order of the projects (which in parts may be different for each build), the fakes generated first were overwritten by the fakes from the following project etc.
Instead of having the fake definitions as part of every project in our solution, we changed it so that we have a solution wide definition of fakes (e.g. System.Web.Fakes.dll) that we link to from each test project where we need it (Add existing item => Add as link). This solved the problems with running the unit tests on the build server, all tests pass again.
I want to get rid of the dialog saying
There were build errors. Would you like to continue and run tests from
the last successful build
when I am running unit tests (I use test -> run -> all tests).
How do I do that? I already know how to disable it when running a normal project.
I also want to know how this can ever be a useful feature?
How do I do that?
You can't.
I also want to know how this can ever be a useful feature?
I find it useful when I am working closely with someone who is configuring test data. I can re-run tests to ensure new test data is valid, without having to worry about compiling any changes that I have done in the meantime.
For example if someone has changed some data in a database, I want to be able to run my tests to ensure that this new data is valid, and I want to be able to run the tests whether the current state of my code compiles or not.
For Visual Studio 2017 / MSTestV2 you can do:
Tools > Options > Project and Solutions > Build and Run
"On Run, when build or deployment errors occur:" "Do not launch"
I have 80+ VS2010 solutions, each contains some unit test projects.
All solutions are merged into one big solution before build process.
All tests (or some test subset) are executed after the successful build.
I want to enable test code coverage for all tests to get exact code coverage for all output assemblies.
My question is: how to enable code coverage without using Visual Studio?
Note: I'm using TFS2010 (MSBuild) to build merged solution. VS 2010 Premium is installed on the build server. MSTest is used for test execution.
You can use JetBrain's TeamCity Professional. It is a CI server that supports executing unit tests and calculating code coverage. It is free for small installations.
I think you need to consider deploying a code coverage tool, see here for a comparison (provided you implement .net).We use NCover, which integrated in our TFS-Build in it's console variant and, although it's not trivial to set it up, we 're very satisfied with it.In this post I had briefly described how we inserted NCoverin our build, this might we useful to you even if you go with another tool.
If you create a Vsmdi file in your large solution (ms test will usually do this for you) you can use this to tell the build which assemblies you want to instrument.
This will only provide code coverage for assemblies that have tests run against them. If you're using testrun.config files to decide which tests you want to run, this should be all you need. The code coverage results should then be published to the build drop location
Edit:
This blog post looks like it covers setting up code coverage
MS Test is killing me.
It is so slow compared to NUnit. I am stuck with it because I need to be able to get Test Results into TFS easy. (Plus it works better with Pex and I am using that too).
But I would really like it to go faster. Even just a bit faster would be nice.
Has anyone made a test runner for MS Test tests that goes faster than the one embedded in Visual Studio?
The Gallio platform provides a test adapter for MSTest. You can then run your tests with any runner supporting Gallio, including built-in runners such as Echo or Icarus, and 3rd party runners such as TD.Net or R#.
The non-GUI test runners are usually slightly faster. Thus you may want to try Echo or the PoSh snap-in.
You should be able to integrate NUnit into TFS with some custom build tasks (which some people have done). If you really wanted to get crazy you could transform NUnit test results into .TRX files and import them directly using the MSTEST command line options
TestDriven.Net is probably the fastest one.
Then goes msTest runner embedded into Visual Studio.
I found that ReSharper test runner is slower than built-in, especially if the solution is big.
At the time I was researching, ReSharper seemed to do a full rebuild of the solution, where msTest was just building changed projects.
So your best choice would be TestDriven.Net
Albacore has an MSTest task.
https://github.com/derickbailey/Albacore/wiki/MSTest-Task
You could write a simple rake file that built your project and then ran the MSTest task.
My recommendation is that, initially, you write a project to only run the albacore MSTest task and time how long that takes.
PS: I wouldn't use Gallio because, in my experience, that test runner takes 6 seconds longer than TestDriven.net to run NUnit tests.
PS 2: My real recommendation is to ditch MSTest for xUnit.net or NUnit
From reasons I won't get into, all our unit tests are using the VSTS test framework. I now want to create an MSBuild script that runs the tests, but I don't want to use mstest.exe from various reasons (it's slower, requires Visual Studio installation everywhere, I need to maintain testrunconfig, etc.)
I've seen that TestDriven.net and TeamCity are able to run VSTS tests 'NUnit style', without using mstest.exe. Are you aware of any standalone command line utility that does this?
You can execute Team System Tests (MSTest) in NUnit if you use a special NUnit Addin that recognizes the MS Test Attributes (TestClass, etc).
Exact Magic Software has an open-source "test-adapter" that can do this.
UPDATE: I've reworked Exact Magic's Msts NUnit Adapter for NUnit 2.5.2.
Download here: http://snippetware.googlecode.com/files/ExactMagic.MstsAdapter.zip
Read more about it here: http://www.bryancook.net/2009/10/mstest-nunit-adapter.html
It seems like TeamCity is simply leveraging Gallio to run VS tests. Gallio appears to have msbuild integration and sounds perfect but after a closer look it seems that it would require a VS install just like MSTest as it appears to depend on MS exes:
The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or
${process:VSTESTHOST.EXE} or
${process:QTAGENT.EXE} or
${process:QTAGENT32.EXE} or
${process:QTDCAGENT.EXE} or
${process:QTDCAGENT32.EXE}'.
Host process exited with code: 0
That being said it sounds like at least one person has got it working:
Christoph De Baene - Running MSTest without Visual Studio
It is possible to run MSTests without installing Visual Studio. See how-do-i-use-mstest-without-visual-studio.
I did this so that I could run my tests as part of my CI process. (I am using CC.NET for my CI solution).
I am in a similar situation as you, in that I want to use TestDriven.NET to get code coverage stats. But, I am running into problems. My first problem is that I am using AssemblyInitialize attributes to initialize a database connection. This isn't supported by NUnit so about half of my tests fail whereas they run fine under MSTest.
So, it seems that translating tests from one test framework to another has pitfalls. If you are aware of that, then go forth, but it might be better to try and keep consistent on one test framework.
We run VSTS tests using msbuild TestToolsTask on a Cruise Control server. This does not use the MSTEST executable -- the condition you ask for -- but does use a variety of TFS dependencies.
Note that we are migrating tests off of the VSTS test framework for NUnit, mostly because we can create extensions for NUnit to perform useful tasks.