Since running Metro apps headlessly is still a gray area: Running a metro app headlessly, I've recently decided to add a native unit test project to my Windows Metro app in hopes that I can find a way to run these unit tests in an automated fashion on the build server. Basically, I'm looking for something similar to MSTest.exe - a utility which is great for running tests from batch files and/or scripts.
In fact, I've tried using the new version of MSTest.exe that comes with VS11 on a generated test .dll, but it fails with the error:
"Unable to load the test container 'test.dll' or one of its dependencies... Error details: Could not load file or assembly file://test.dll' or one of its dependencies. The Module was expected to contain an assembly manifest."
Does MSTest.exe work with test containers that contain WinRT code? If not, is there a utility that will do what I want?
Edit: I just found out that MSTest does not support running tests on a Metro style app. Found here: http://msdn.microsoft.com/en-us/library/ms253138%28v=vs.110%29.aspx This really is too bad. I'm still hoping there's a utility out there that will work.
After blindly digging through the VS folders, I happened to find a new test runner under:
C:\Program Files\Microsoft Visual Studio
11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
This utility allows you to execute WinRT unit tests from the command line. It's very similar to MSTest.exe.
There doesn't seem to be any documentation out there for this yet, but at least a help command exists.
If you are executing vstest.console.exe <filename>.dll then your tests do not get executed in appcontainer mode. You need to give <filename>.appx to execute tests in appcontainer mode. More info on how to execute tests for Windows Metro style apps from command line can be found at Running Unit Tests for Windows Metro style apps from Command Line
There is a documentation about (among other things) running unit tests for Windows Phone:
http://msdn.microsoft.com/en-us/library/windowsphone/develop/dn168930(v=vs.105).aspx
It describes also the command line way of doing it using vstest.console.exe.
It also gives a comparison of supported features between unit tests for Windows Phone and WinRT.
One important detail is that unit tests for WinRT cannot be run on a device. This is a pity and relevant to question How to automate non-interactive tests on Microsoft Surface
Related
We have a C++ Linux project in Visual Studio that we build and run remotely on a Docker container. I am beginning to write unit tests for the project using Google Test (not tied to this framework) which I also want to run remotely. Unfortunately the tests are run locally which doesn't suit what I am trying to achieve. Does anyone know if this can be done with any test framework for C++ and if so could you point me to relevant docs etc?
I know I can create another standard Linux project containing the tests which I can run remotely but then I lose the Test Explorer functionality built-in to VS which I would prefer to be able to use.
Any help here would be greatly appreciated.
Currently how I run my unit tests:
I have a visual studio project, which builds a C++ google test (unit test) executable.
I then copy this exe to my Virtual Machine with QNX and run it manually, to get the unit test results.
Can I use resharper to automate this?
Resharper is able to detect and list tests from test file. But on running them, it throws error as it is not able to execute the tests in Windows.
Below is the exact error, I get, on running any of the tests.
ReSharper Ultimate – Error while running unit tests:
Invalid path
Can I set up resharper and Visual studio environment, so that, I can run tests directly from visual studio, which will copy the exe to the Virtual Machine and give me the execution results?
This will avoid me manually copy pasting the exe and running command to execute the tests?
I searched a lot regarding this in google. But couldn't find much help.
I'm new to google test and resharper. So any help would be greatly appreciated.
Thanks in advance.
No, I'm afraid ReSharper cannot do that out of the box.
Though it is quite powerful, so if you manage to write a script that does the VM specific part, ReSharper should be able to cope with it. What that script needs to do is:
It must be run on the host (Windows).
Copy the test binary to the QNX system in the VM.
Execute the test.
Capture the test's output and redirect it to the host's stdout.
The ReSharper options can be set here:
Menu ReSharper > Options...
In the left pane scroll down to Tools > Unit Testing > C++ Tests
In the text field Command: you need to enter the path to the script.
See https://www.jetbrains.com/help/resharper/Reference_Options_Tools_Unit_Testing_CPP_Tests.html for more details.
I think you'll find that the msg box showing "ReSharper Ultimate – Error while running unit tests: Invalid path" has been caused by the last update to VS 2017, which has broken things. I usually run my google tests through resharper c++, and until a few days ago, it worked. Now, I see this exact same error.
I'm coming from a native iOS / Android development background and I'm trying to understand the tooling around Xamarin Unit Testing using the Command Line.
From my point of view there are two types of code that you want to Unit Test:
Plain Old C# Code - with no dependencies to any iOS / Android framework - so it shouldn't need an iOS / Android emulator to run on
Code that depends on iOS / Android frameworks that needs to run on a device / emulator
The official Xamarin documentation mentions NUnitLite / Touch.Unit but it doesn't mention any support around Command Line. I did found an example though, but it's not clear to me if this is a tool that's officially supported by Xamarin. Also it seems that you can run tests only on the emulator/device using that tool.
Another example I've found around refers to xUnit.net - it seems that you can also run tests without an emulator / device, and that you can also run them on an emulator / device - however in that specific blogpost it's not documented how you do that.
So my question is: How should I approach Xamarin Unit Testing and what tools do you recommend using so I can have Command Line support in my CI.
Thank you
The most popular unit testing frameworks used with Xamarin are NUnit and XUnit. They are both similar to JUnit.
Usually a Xamarin cross-platform app uses a Portable Class Library (PCL) project where the platform agnostic (shared) code sits: business logic, model, view models, service etc. This code is unit tested in a seperate pcl or.net45 test project which references the source project and nunit/xunit.
To run the nunit/xunit unit tests you need to run the corresponding test runner and point it tou your test assembly. Both nunit and xundit feature console runners which can be parameterized at will from your command line (see links).
Feel free to chose either nunit or xunit. I like them both.
You might also have platform specific unit tests (which depend on the android/ios/uwp sdks) and that have to be run on a device. These tests can also be created with nunit or xunit and run with nunit device runner or xunit device runner. Basically what will happen here is you add an android/ios app project for testing which references nunit/junit, contains your device specific tests and links to your shared tests and can run them both on the device.
There is also the layer of coded UI tests where NUnit, Xamarin UITest and Specflow might be of use. Im guessing this part is beyond the scope of your question.
But then again you are coming form Android and are used with gradle. Well Xamarin and .net does not have gradle but it has Cake. I use it to automate all my project builds/tests/ci/deployments etc.
Cake (C# Make) is a cross platform build automation system with a C# DSL to do things like compiling code, copy files/folders, running unit tests, compress files and build NuGet packages.
Your Cake script can look something like this:
Task("Run-Unit-Tests")
.IsDependentOn("Build")
.Does(() =>
{
NUnit("./src/**/bin/" + configuration + "/*.Tests.dll");
});
Task("Build")
.Does(() =>
{
DotNetBuild("YourAndroid.csproj");
DotNetBuild("YourCoreTests.csproj");
...
}
);
Cake comes with a bootstrapper file either (ps1 - powershell for windows or sh for mac) which downloads all the tools you need to run your script (cake itself, nuget, nunit/xunit runner etc).
Your command line/CI can run it like this:
./build.sh -Target Run-Unit-Tests
Unit testing in painful on Xamarin.
Don't have huge experience in Unit Testing mobile project, but if you want to test the app I would recommend this integration test approach:
1) For any calculation type functionality (you called it "Plain Old C#") use NUnit (it is supported by mono).
However I cannot come up with an example of such code, as heavy calculations should be done on server side. And there you do separate Unit tests
2) UITests(NUnit again) can be done to prove app is working and UI interaction call needed behavior.
Xamarin also providing TestCloud where app could be tested on many devices, but it is paid for Service. As Alternative you can setup build server like Jenkins to do this job for you.
Anyway - it is my view on how this could be done and hope it answered a bit on your question.
I created a test framework to test a sample application. This basically required creating a Solution for my sample application and then adding a Native Test Project, which can test the application.
The test project generates an .exe, but if I were to execute the .exe on it's own, it does nothing.
Is there a way to get information from the test programme so it can be used for other purposes outside of the VS environment?
Yes, you can run unit tests without VS2013.
Use Run automated tests from the command line using MSTest source to do so.
In general it says:
Open a Visual Studio command prompt.
Either change directory to your solution folder or, when you run the MSTest.exe program in step 3, specify a full or relative path to the metadata file or to the test container.
Run the MSTest.exe program.
I hope it will solve your issue.
It took a couple of days, but I've found that DeJaVo's answer is incorrect. Because I am running a Native Test Project, MSTest will not work with it. To run a Native Test from the command line, use VSTest.Console.
From reasons I won't get into, all our unit tests are using the VSTS test framework. I now want to create an MSBuild script that runs the tests, but I don't want to use mstest.exe from various reasons (it's slower, requires Visual Studio installation everywhere, I need to maintain testrunconfig, etc.)
I've seen that TestDriven.net and TeamCity are able to run VSTS tests 'NUnit style', without using mstest.exe. Are you aware of any standalone command line utility that does this?
You can execute Team System Tests (MSTest) in NUnit if you use a special NUnit Addin that recognizes the MS Test Attributes (TestClass, etc).
Exact Magic Software has an open-source "test-adapter" that can do this.
UPDATE: I've reworked Exact Magic's Msts NUnit Adapter for NUnit 2.5.2.
Download here: http://snippetware.googlecode.com/files/ExactMagic.MstsAdapter.zip
Read more about it here: http://www.bryancook.net/2009/10/mstest-nunit-adapter.html
It seems like TeamCity is simply leveraging Gallio to run VS tests. Gallio appears to have msbuild integration and sounds perfect but after a closer look it seems that it would require a VS install just like MSTest as it appears to depend on MS exes:
The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or
${process:VSTESTHOST.EXE} or
${process:QTAGENT.EXE} or
${process:QTAGENT32.EXE} or
${process:QTDCAGENT.EXE} or
${process:QTDCAGENT32.EXE}'.
Host process exited with code: 0
That being said it sounds like at least one person has got it working:
Christoph De Baene - Running MSTest without Visual Studio
It is possible to run MSTests without installing Visual Studio. See how-do-i-use-mstest-without-visual-studio.
I did this so that I could run my tests as part of my CI process. (I am using CC.NET for my CI solution).
I am in a similar situation as you, in that I want to use TestDriven.NET to get code coverage stats. But, I am running into problems. My first problem is that I am using AssemblyInitialize attributes to initialize a database connection. This isn't supported by NUnit so about half of my tests fail whereas they run fine under MSTest.
So, it seems that translating tests from one test framework to another has pitfalls. If you are aware of that, then go forth, but it might be better to try and keep consistent on one test framework.
We run VSTS tests using msbuild TestToolsTask on a Cruise Control server. This does not use the MSTEST executable -- the condition you ask for -- but does use a variety of TFS dependencies.
Note that we are migrating tests off of the VSTS test framework for NUnit, mostly because we can create extensions for NUnit to perform useful tasks.