Currently I am experimenting with Mono, and creating a small testproject. When I tried to write some unittests I recognized that there is no default unittest framework like in VS. I decided to go with nUnit (but mbUnit, XUnit,...) would be fine too.
My problem is, that I was not able to find a testrunner that executes my tests and displays the testresults.
Is there a good tool available (currently I am using Monodevelop 2.4)? It is not a requirement tha it is integrated within monodevelop, so an external tool would be good enough.
NUnit is fine. I am using it with mono and works perfectly. For test runner you can use
nunit-console [input]
or the gui one
nunit-gui
These apps can be probably installed from your distribution repository (Ubuntu has it for sure) or manually installed using binaries from NUnit site. As you have written, you used to use built in VS tests. If you are using resharper, nunit will integrate as well, I also find it very clean.
I'd like to add that from the gui runner is winforms application and does not look (and feel ;)) too good on my OS. Nevertheless works fine.
Related
I'm coming from a native iOS / Android development background and I'm trying to understand the tooling around Xamarin Unit Testing using the Command Line.
From my point of view there are two types of code that you want to Unit Test:
Plain Old C# Code - with no dependencies to any iOS / Android framework - so it shouldn't need an iOS / Android emulator to run on
Code that depends on iOS / Android frameworks that needs to run on a device / emulator
The official Xamarin documentation mentions NUnitLite / Touch.Unit but it doesn't mention any support around Command Line. I did found an example though, but it's not clear to me if this is a tool that's officially supported by Xamarin. Also it seems that you can run tests only on the emulator/device using that tool.
Another example I've found around refers to xUnit.net - it seems that you can also run tests without an emulator / device, and that you can also run them on an emulator / device - however in that specific blogpost it's not documented how you do that.
So my question is: How should I approach Xamarin Unit Testing and what tools do you recommend using so I can have Command Line support in my CI.
Thank you
The most popular unit testing frameworks used with Xamarin are NUnit and XUnit. They are both similar to JUnit.
Usually a Xamarin cross-platform app uses a Portable Class Library (PCL) project where the platform agnostic (shared) code sits: business logic, model, view models, service etc. This code is unit tested in a seperate pcl or.net45 test project which references the source project and nunit/xunit.
To run the nunit/xunit unit tests you need to run the corresponding test runner and point it tou your test assembly. Both nunit and xundit feature console runners which can be parameterized at will from your command line (see links).
Feel free to chose either nunit or xunit. I like them both.
You might also have platform specific unit tests (which depend on the android/ios/uwp sdks) and that have to be run on a device. These tests can also be created with nunit or xunit and run with nunit device runner or xunit device runner. Basically what will happen here is you add an android/ios app project for testing which references nunit/junit, contains your device specific tests and links to your shared tests and can run them both on the device.
There is also the layer of coded UI tests where NUnit, Xamarin UITest and Specflow might be of use. Im guessing this part is beyond the scope of your question.
But then again you are coming form Android and are used with gradle. Well Xamarin and .net does not have gradle but it has Cake. I use it to automate all my project builds/tests/ci/deployments etc.
Cake (C# Make) is a cross platform build automation system with a C# DSL to do things like compiling code, copy files/folders, running unit tests, compress files and build NuGet packages.
Your Cake script can look something like this:
Task("Run-Unit-Tests")
.IsDependentOn("Build")
.Does(() =>
{
NUnit("./src/**/bin/" + configuration + "/*.Tests.dll");
});
Task("Build")
.Does(() =>
{
DotNetBuild("YourAndroid.csproj");
DotNetBuild("YourCoreTests.csproj");
...
}
);
Cake comes with a bootstrapper file either (ps1 - powershell for windows or sh for mac) which downloads all the tools you need to run your script (cake itself, nuget, nunit/xunit runner etc).
Your command line/CI can run it like this:
./build.sh -Target Run-Unit-Tests
Unit testing in painful on Xamarin.
Don't have huge experience in Unit Testing mobile project, but if you want to test the app I would recommend this integration test approach:
1) For any calculation type functionality (you called it "Plain Old C#") use NUnit (it is supported by mono).
However I cannot come up with an example of such code, as heavy calculations should be done on server side. And there you do separate Unit tests
2) UITests(NUnit again) can be done to prove app is working and UI interaction call needed behavior.
Xamarin also providing TestCloud where app could be tested on many devices, but it is paid for Service. As Alternative you can setup build server like Jenkins to do this job for you.
Anyway - it is my view on how this could be done and hope it answered a bit on your question.
I have a Windows 8 Store App with two projects within the solution - one project exclusively for tests. I have added NUnit and NUnit Test Adapter (https://www.nuget.org/packages/NUnitTestAdapter/1.0.0) for this through nuget.
My tests are detected - but I cannot run or debug them. Looking at the test output window, I get the following error:
Could not find test executor with URI 'executor://nunittestexecutor/'. Make sure that the test executor is installed and supports .net runtime version 4.0.30319.34003.
I am not using resharper and I am using Visual Studio Professional 2012 Update 3 on Windows 8.1. References for the test project are as follows:
Based on my comments above, below is a workaround. Unfortunately at this stage, Windows 8.1 store Unit Test project types, using NUnit extension wouldn't work due to the different .NET targets. I tried with different Test Unit Adapters including an NUnitTestAdapterWithFramework.
It seems that the issue you haveing was occurring with standard .NET libraries targeting NUnit test adapter but the above NUnitTestAdapterWithFramework must have fixed those issue. See the Q & A section of the NUnitTestExtension
But unfortunatly it seems that this still of an issue that hasn't been fixed for Win8 Store App type Unit Testing. Pretty sure xUnit.NET also not compatible yet with different .NET target types (i,e WinRT)
So what are the options?
a. For your group, you can change them to use MSTest framework. Outcome - Problem Solved no issues.
b. Workaround "linked project". Outcome - Can't *guarantee** but this should also work.
With option 'b'
In your comment you mentioned.
but I'm still not sure what it does or how to implement a 'linked
project', do you have any more information on this? Also, as this is
for a group university project, I was hoping i wouldn't have to force
too many workarounds
When you think about it, it is not really hard work around. It is simple and I'm sure your group would be able to apply this workaround easily.
Please follow the below steps.
Create a separate class library in your solution (you can target .NET framework 4).
Then add NUnit assemblies and the NUnit test adapter as usual.
Right click on this project and select 'Add' then 'Existing Item'
Select the Win8 Store Unit Test project and locate the Unit Test file you want to add. When you add the file, make sure you select 'Add As a Link' button. Please see below.
Now rebuild the solution, close and re-open the UnitTest explorer and you should be able to run those tests.
*The reason I said can't guaranteed. I haven't really written Unit tests against Win8 App. So if your SUT (System Under Test) require special configuration it might cause issues. But I'm not sure.
Finally creating a link files are not that hard if everything works you can continue to do this until NUnit has the support for Win8 Unit Testing. Or the other option is simply change all your Unit Tests to use MSTest framework if possible.
Is there actually a good way to unit test MonoTouch projects using NUnit and the MonoDevelop test runner?
I know there is the official MonoTouch unit tests project type, but running tests within the simulator isn't the way I want to go. For now I want to run tests with the MonoDevelop test runner, later everything should work with Jenkins (CI).
I know the limitations about UI specific code, so everything I want to test has nothing to do with MonoTouch itself, it's all about business logic placed within separete projects.
By adding tests to MonoTouch Library type projects, I am getting System.IO.FileNotFoundException's as described here: http://ben.phegan.name/index.php/2011/02/28/monotouch-and-unit-testing/
By using a separate NUnit test project, I can't reference my system under test, because its project type is of type MonoTouch library project, which, of course, has an incompatible target framework (vMonoTouch).
So, there isn't any real alternative to Touch.Unit, is it?
Is there actually a good way to unit test MonoTouch projects using NUnit
Touch.Unit
and the MonoDevelop test runner?
Not really. MonoTouch projects depends on monotouch.dll which needs to execute under iOS (not OSX). So there's a need for the runner to execute on the simulator or devices.
Now there's a few misconceptions in your question:
later everything should work with Jenkins (CI).
Touch.Unit is already used with continuous builds / integration servers (as long as they are running OSX) using both the iOS simulator and/or devices. Details are available here.
I know the limitations about UI specific code,
Touch.Unit is not about UI testing. In fact it's pretty bad at UI testing (but that's beside the point).
Touch.Unit is a test runner that execute on iOS. That allows you to use MonoTouch / iOS API inside your own tests (it could be UIKit, but it could be StoreKit, GameKit, *Kit, any Foundation class... it's a quite large world).
So, there isn't any real alternative to Touch.Unit, is it?
Yes. If your business logic is well isolated and does not depend on monotouch.dll then you should be able to build it either as:
a non-MonoTouch project (different project, same sources), i.e. linked with the regular framework; or
link to the sources from within your unit test assembly (which links to the regular framework);
That classic nunit test assembly would then be a regular framework project and will be able to run from the default NUnit runner or from within MonoDevelop unit test runner.
We're using JIRA with Bamboo as build server for continuous integration.
But I have a native C++ project (using Visual Studio 2010), and apparently Microsoft doesn't support unit tests for native C++ code. There's some tools like WinUnit or cfix that seem to do the job, but neither do I know about their compatibility to Bamboo, nor do I know which tool is the best / easiest to use / has the most features / has best VS compatibility ...
Does someone have experience with that?
Just as a heads up, my current project is using c++ with bamboo for CI. We use googletest for our testing framework. If you have the build run with --gtest_output=xml:{file or Directory} it will produce xml output that can parsed by Bamboo's JUnit Parse task.
I played a bit with CppUnit. The way it works is that you create an executable project which you fill with your test cases. When run, it runs your tests. I think cfix does that but I've never used it for user mode testing. To run the tests as part of your build process, put in a post build step for your test project that runs the test exe. It'll run when you build your Visual Studio solution.
From reasons I won't get into, all our unit tests are using the VSTS test framework. I now want to create an MSBuild script that runs the tests, but I don't want to use mstest.exe from various reasons (it's slower, requires Visual Studio installation everywhere, I need to maintain testrunconfig, etc.)
I've seen that TestDriven.net and TeamCity are able to run VSTS tests 'NUnit style', without using mstest.exe. Are you aware of any standalone command line utility that does this?
You can execute Team System Tests (MSTest) in NUnit if you use a special NUnit Addin that recognizes the MS Test Attributes (TestClass, etc).
Exact Magic Software has an open-source "test-adapter" that can do this.
UPDATE: I've reworked Exact Magic's Msts NUnit Adapter for NUnit 2.5.2.
Download here: http://snippetware.googlecode.com/files/ExactMagic.MstsAdapter.zip
Read more about it here: http://www.bryancook.net/2009/10/mstest-nunit-adapter.html
It seems like TeamCity is simply leveraging Gallio to run VS tests. Gallio appears to have msbuild integration and sounds perfect but after a closer look it seems that it would require a VS install just like MSTest as it appears to depend on MS exes:
The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or
${process:VSTESTHOST.EXE} or
${process:QTAGENT.EXE} or
${process:QTAGENT32.EXE} or
${process:QTDCAGENT.EXE} or
${process:QTDCAGENT32.EXE}'.
Host process exited with code: 0
That being said it sounds like at least one person has got it working:
Christoph De Baene - Running MSTest without Visual Studio
It is possible to run MSTests without installing Visual Studio. See how-do-i-use-mstest-without-visual-studio.
I did this so that I could run my tests as part of my CI process. (I am using CC.NET for my CI solution).
I am in a similar situation as you, in that I want to use TestDriven.NET to get code coverage stats. But, I am running into problems. My first problem is that I am using AssemblyInitialize attributes to initialize a database connection. This isn't supported by NUnit so about half of my tests fail whereas they run fine under MSTest.
So, it seems that translating tests from one test framework to another has pitfalls. If you are aware of that, then go forth, but it might be better to try and keep consistent on one test framework.
We run VSTS tests using msbuild TestToolsTask on a Cruise Control server. This does not use the MSTEST executable -- the condition you ask for -- but does use a variety of TFS dependencies.
Note that we are migrating tests off of the VSTS test framework for NUnit, mostly because we can create extensions for NUnit to perform useful tasks.