Without an IDE integration, running F# unit tests is pretty cumbersome.
One needs to compile them and then pass the resulting dll to a test runner.
I want to find an easier way to run the kind of tests you find for problems on http://exercism.io/
Any ideas?
Related
Is there actually a good way to unit test MonoTouch projects using NUnit and the MonoDevelop test runner?
I know there is the official MonoTouch unit tests project type, but running tests within the simulator isn't the way I want to go. For now I want to run tests with the MonoDevelop test runner, later everything should work with Jenkins (CI).
I know the limitations about UI specific code, so everything I want to test has nothing to do with MonoTouch itself, it's all about business logic placed within separete projects.
By adding tests to MonoTouch Library type projects, I am getting System.IO.FileNotFoundException's as described here: http://ben.phegan.name/index.php/2011/02/28/monotouch-and-unit-testing/
By using a separate NUnit test project, I can't reference my system under test, because its project type is of type MonoTouch library project, which, of course, has an incompatible target framework (vMonoTouch).
So, there isn't any real alternative to Touch.Unit, is it?
Is there actually a good way to unit test MonoTouch projects using NUnit
Touch.Unit
and the MonoDevelop test runner?
Not really. MonoTouch projects depends on monotouch.dll which needs to execute under iOS (not OSX). So there's a need for the runner to execute on the simulator or devices.
Now there's a few misconceptions in your question:
later everything should work with Jenkins (CI).
Touch.Unit is already used with continuous builds / integration servers (as long as they are running OSX) using both the iOS simulator and/or devices. Details are available here.
I know the limitations about UI specific code,
Touch.Unit is not about UI testing. In fact it's pretty bad at UI testing (but that's beside the point).
Touch.Unit is a test runner that execute on iOS. That allows you to use MonoTouch / iOS API inside your own tests (it could be UIKit, but it could be StoreKit, GameKit, *Kit, any Foundation class... it's a quite large world).
So, there isn't any real alternative to Touch.Unit, is it?
Yes. If your business logic is well isolated and does not depend on monotouch.dll then you should be able to build it either as:
a non-MonoTouch project (different project, same sources), i.e. linked with the regular framework; or
link to the sources from within your unit test assembly (which links to the regular framework);
That classic nunit test assembly would then be a regular framework project and will be able to run from the default NUnit runner or from within MonoDevelop unit test runner.
In Visual Studio 11, I'm facing a complex issue.
I'm using the Unit Test Explorer to run unit-tests. However, it shows both my normal unit tests and my ordered tests, which are basically collections of tests to run in a specific order too.
That doesn't work well for me, because I would love to have the setting "Run tests after build" on. However, as you would expect, what happens is that it runs both all my tests individually out of order and then runs my ordered test as shown in the picture below.
I would really love to be able to run only my ordered tests when the build finishes. I am not interested in third-party plugins/extensions to help me do this.
I ended up using a build server, and then configured my build configuration as follows, which will run all tests with the orderedtest file extension:
I have 80+ VS2010 solutions, each contains some unit test projects.
All solutions are merged into one big solution before build process.
All tests (or some test subset) are executed after the successful build.
I want to enable test code coverage for all tests to get exact code coverage for all output assemblies.
My question is: how to enable code coverage without using Visual Studio?
Note: I'm using TFS2010 (MSBuild) to build merged solution. VS 2010 Premium is installed on the build server. MSTest is used for test execution.
You can use JetBrain's TeamCity Professional. It is a CI server that supports executing unit tests and calculating code coverage. It is free for small installations.
I think you need to consider deploying a code coverage tool, see here for a comparison (provided you implement .net).We use NCover, which integrated in our TFS-Build in it's console variant and, although it's not trivial to set it up, we 're very satisfied with it.In this post I had briefly described how we inserted NCoverin our build, this might we useful to you even if you go with another tool.
If you create a Vsmdi file in your large solution (ms test will usually do this for you) you can use this to tell the build which assemblies you want to instrument.
This will only provide code coverage for assemblies that have tests run against them. If you're using testrun.config files to decide which tests you want to run, this should be all you need. The code coverage results should then be published to the build drop location
Edit:
This blog post looks like it covers setting up code coverage
MS Test is killing me.
It is so slow compared to NUnit. I am stuck with it because I need to be able to get Test Results into TFS easy. (Plus it works better with Pex and I am using that too).
But I would really like it to go faster. Even just a bit faster would be nice.
Has anyone made a test runner for MS Test tests that goes faster than the one embedded in Visual Studio?
The Gallio platform provides a test adapter for MSTest. You can then run your tests with any runner supporting Gallio, including built-in runners such as Echo or Icarus, and 3rd party runners such as TD.Net or R#.
The non-GUI test runners are usually slightly faster. Thus you may want to try Echo or the PoSh snap-in.
You should be able to integrate NUnit into TFS with some custom build tasks (which some people have done). If you really wanted to get crazy you could transform NUnit test results into .TRX files and import them directly using the MSTEST command line options
TestDriven.Net is probably the fastest one.
Then goes msTest runner embedded into Visual Studio.
I found that ReSharper test runner is slower than built-in, especially if the solution is big.
At the time I was researching, ReSharper seemed to do a full rebuild of the solution, where msTest was just building changed projects.
So your best choice would be TestDriven.Net
Albacore has an MSTest task.
https://github.com/derickbailey/Albacore/wiki/MSTest-Task
You could write a simple rake file that built your project and then ran the MSTest task.
My recommendation is that, initially, you write a project to only run the albacore MSTest task and time how long that takes.
PS: I wouldn't use Gallio because, in my experience, that test runner takes 6 seconds longer than TestDriven.net to run NUnit tests.
PS 2: My real recommendation is to ditch MSTest for xUnit.net or NUnit
From reasons I won't get into, all our unit tests are using the VSTS test framework. I now want to create an MSBuild script that runs the tests, but I don't want to use mstest.exe from various reasons (it's slower, requires Visual Studio installation everywhere, I need to maintain testrunconfig, etc.)
I've seen that TestDriven.net and TeamCity are able to run VSTS tests 'NUnit style', without using mstest.exe. Are you aware of any standalone command line utility that does this?
You can execute Team System Tests (MSTest) in NUnit if you use a special NUnit Addin that recognizes the MS Test Attributes (TestClass, etc).
Exact Magic Software has an open-source "test-adapter" that can do this.
UPDATE: I've reworked Exact Magic's Msts NUnit Adapter for NUnit 2.5.2.
Download here: http://snippetware.googlecode.com/files/ExactMagic.MstsAdapter.zip
Read more about it here: http://www.bryancook.net/2009/10/mstest-nunit-adapter.html
It seems like TeamCity is simply leveraging Gallio to run VS tests. Gallio appears to have msbuild integration and sounds perfect but after a closer look it seems that it would require a VS install just like MSTest as it appears to depend on MS exes:
The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or
${process:VSTESTHOST.EXE} or
${process:QTAGENT.EXE} or
${process:QTAGENT32.EXE} or
${process:QTDCAGENT.EXE} or
${process:QTDCAGENT32.EXE}'.
Host process exited with code: 0
That being said it sounds like at least one person has got it working:
Christoph De Baene - Running MSTest without Visual Studio
It is possible to run MSTests without installing Visual Studio. See how-do-i-use-mstest-without-visual-studio.
I did this so that I could run my tests as part of my CI process. (I am using CC.NET for my CI solution).
I am in a similar situation as you, in that I want to use TestDriven.NET to get code coverage stats. But, I am running into problems. My first problem is that I am using AssemblyInitialize attributes to initialize a database connection. This isn't supported by NUnit so about half of my tests fail whereas they run fine under MSTest.
So, it seems that translating tests from one test framework to another has pitfalls. If you are aware of that, then go forth, but it might be better to try and keep consistent on one test framework.
We run VSTS tests using msbuild TestToolsTask on a Cruise Control server. This does not use the MSTEST executable -- the condition you ask for -- but does use a variety of TFS dependencies.
Note that we are migrating tests off of the VSTS test framework for NUnit, mostly because we can create extensions for NUnit to perform useful tasks.