I'm building a R# plugin and decided to cover it with tests. However, I noticed, that plugin tests cannot be run using the R# runner if plugin target version is the same as R# version.
For instance, if you try to run tests using JetBrains.ReSharper.SDK.Tests of version 2016.2.20160818.171542, you'll get the following error:
ERROR System.TypeLoadException:
Method 'RunTests' in type 'JetBrains.ReSharper.UnitTestRunner.nUnit26.BuiltInNUnitRunner' from assembly 'JetBrains.ReSharper.UnitTestRunner.nUnit26, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' does not have an implementation.
In previous versions the error was different.
Likely, that happens because R# runner and SDK assemblies make a mix.
Is there any way to configure project/R# to be able to run plugin unit tests using the R# runner? Likely, R# dev team uses their own runner.
Thanks!
Unfortunately, this is a bug in the ReSharper runner in 201.6.2. Something to do with having multiple copies of the same assemblies in the same process - the assemblies from the install, and the assemblies from the bin folder. Something's getting messed up along the way.
You can subscribe to, vote and track this issue: RSRP-460399
The workaround is to install an older version of ReSharper into a Visual Studio "experimental instance" aka "custom hive", and use this instance to develop and test the plugin. The normal instance can be used for other day-to-day development.
You can download an older version of ReSharper here (e.g. 2016.1.2), and in the installer, click Options and enter the name of a custom instance, e.g. "Plugins". Then create a new shortcut to Visual Studio and set the command line to "devenv.exe /rootSuffix Plugins" (or whatever you called the custom hive).
R# and SDK must be either of different major versions, or of binary compatible versions. Unfortunately, within EAP the internal APIs inside the test runner have changed, so R# and SDK from different EAPs are not compatible.
Related
I upgraded to Visual Studio 2017 last week. A solution with several test projects using MSTest v1 seemed to be fine at first. Most test projects execute tests fine. This is .NET Framework, not .NET Core.
However, there is one project where I am encountering issues. It's an integration test project, meaning it runs web application tests using the ChromeDriver and Selenium. This project references NUnit, though that reference comes from another project reference. I am not actually using NUnit as my test framework.
This is the error when trying to run any test in that particular test project:
Could not find test executor with URI 'executor://mstestadapter/v1'. Make sure that the test executor is installed and supports .net runtime version .
Test project {name} does not reference any .NET NuGet adapter. Test discovery or execution might not work for this project.
It's recommended to reference NuGet test adapters in each test project in the solution.
Disabling the option indicated below made everything work:
I found this solution after browsing the preview release notes at https://learn.microsoft.com/en-us/visualstudio/releasenotes/vs2017-preview-relnotes#--visual-studio-2017-version-158-preview-4 (scroll down to "... What's New ..." and then .NET Test Adapter Extension). It mentions this breaking change (though it didn't break my other projects), but there is no mention of what to actually do to solve the problem.
Of course, long-term, keeping that option unchecked doesn't seem feasible. So I am looking for guidance on how to proceed.
I already tried to see if I could "quickly" upgrade to MSTest v2, but that had an issue because System.Runtime 4.1.2 couldn't be found. I tried adding System.Runtime 4.3.0 via NuGet, but that didn't solve the issue.
I was getting a similar issue running Xamarin.UITests and resolved it by installing the "NUnit 2 Test Adapter" Extension in Visual Studio. In Visual Studio 2017 you do this from Tools > > Extensions and Updates... > Online > Search "NUnit Adapter" > click the NUnit 2 Test Adapter then the Download button to install. Restart Visual Studio when prompted to complete the installation.
It would be interesting to understand how the MSTest V1 framework assembly(Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll) is referenced in this project. The compatibility checkbox that also ended up being disabled in the image above should ensure that this project has MSTestV1 tests and enabled run tests. If it isn't then for some reason the detection logic is busted and needs to be fixed - a snippet on how the assembly is referenced would be helpful there.
As for moving to MSTest V2, this blog should help: https://blogs.msdn.microsoft.com/devops/2016/06/17/taking-the-mstest-framework-forward-with-mstest-v2/ . And for the specific issue you seem to be running into, this github issue on the repo sounds similar: https://github.com/Microsoft/testfx/issues/184 .
Caltor's answer pretty much solved it for me: it was the NUnit Adapter I needed. But I followed slightly different steps. Here they are for VS 2019 (but probably apply to VS 17 too):
Right clicked on the References node under my project in the solution explorer
Chose "Manager NuGet packages"
Clicked Browse
Searched for nunit3testadapter
Hit install on the result that popped up
The issue for me was similar to the OP's - the adapter was installed on some projects but not others. But confusingly this wasn't immediately visible under the "References" node for that project. The only node relating to nunit under that References node said "nunit.framework". So the references for test with and without the adapter looked the same.
"... but that had an issue because System.Runtime 4.1.2 couldn't be found."
I think this is a setting in the app.config that does not get updated when the project settings are updated. I searched for System.Runtime under the packages folder and then changed my app.config to match what I found, and then that particular problem went away.
Environment:
.Net Core 2.0
visual Studio Professional 2017
NUnit3
OpenCover (For code coverage)
ReportGenerator(For code coverage report)
Hi Nunit Team,
I am pretty familiar with MS Test framework and have worked on that extensively but was testing how suitable NUnit framework would be and so wanted to do a small POC around that. I created a small classlibrary and a unit test project. Below are the NuGet package been installed:
NUnit 3.7.1
But when I try to run the test cases from console runner like below I get exception saying System.Runtime, Version=4.2.0.0 assembly not found.
C:\Users\UN\.nuget\packages\nunit.consolerunner\3.8.0\tools>nunit3-console.exe D:\Code_Trials\BoMLibrary\BomTest\bin\Debug\netcoreapp2.0\BomTest.dll
Exception detail:
Errors, Failures and Warnings
Invalid : D:\Code_Trials\BoMLibrary\BomTest\bin\Debug\netcoreapp2.0\BomTest.dll
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
----> Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Run Settings
DisposeRunners: True
WorkDirectory: C:\Users\Smukherjee.nuget\packages\nunit.consolerunner\3.8.0\tools
ImageRuntimeVersion: 4.0.30319
ImageTargetFrameworkName: .NETCoreApp,Version=v2.0
ImageRequiresX86: False
ImageRequiresDefaultAppDomainAssemblyResolver: False
NumberOfTestWorkers: 4
Am really not sure why so? No matter what I try not able to pass through this error. Moreover, when I try to install the assembly from Nuget I don't even find a assembly with version 4.2.0.0.
Tried installing the version 4.3.0.0 of System.Runtime but no luck. Can you please help on this.
Strange thing, I have got a sample unit test project (as part of opencover nuget install) which targets .Net Framework 4.5 and have NUnit2.6.7 package installed and when I try to run that using NUnit3 console it works like charm.
Even, installing NUnit3 latest stable version 3.10.1.0 doesn't help either. Also, it's surely not an Opencover issue since opencover console just passes the test run request to NUnit console. So the issue has to be around NUnit.
Let me know if you need more information around this.
Found the answer to this here https://github.com/nunit/nunit/issues/2763
NUnit Console doesn't run .NET Core tests - you need to use dotnet test instead. Take a look at this page:
https://github.com/nunit/docs/wiki/.NET-Core-and-.NET-Standard
I have a Windows 8 Store App with two projects within the solution - one project exclusively for tests. I have added NUnit and NUnit Test Adapter (https://www.nuget.org/packages/NUnitTestAdapter/1.0.0) for this through nuget.
My tests are detected - but I cannot run or debug them. Looking at the test output window, I get the following error:
Could not find test executor with URI 'executor://nunittestexecutor/'. Make sure that the test executor is installed and supports .net runtime version 4.0.30319.34003.
I am not using resharper and I am using Visual Studio Professional 2012 Update 3 on Windows 8.1. References for the test project are as follows:
Based on my comments above, below is a workaround. Unfortunately at this stage, Windows 8.1 store Unit Test project types, using NUnit extension wouldn't work due to the different .NET targets. I tried with different Test Unit Adapters including an NUnitTestAdapterWithFramework.
It seems that the issue you haveing was occurring with standard .NET libraries targeting NUnit test adapter but the above NUnitTestAdapterWithFramework must have fixed those issue. See the Q & A section of the NUnitTestExtension
But unfortunatly it seems that this still of an issue that hasn't been fixed for Win8 Store App type Unit Testing. Pretty sure xUnit.NET also not compatible yet with different .NET target types (i,e WinRT)
So what are the options?
a. For your group, you can change them to use MSTest framework. Outcome - Problem Solved no issues.
b. Workaround "linked project". Outcome - Can't *guarantee** but this should also work.
With option 'b'
In your comment you mentioned.
but I'm still not sure what it does or how to implement a 'linked
project', do you have any more information on this? Also, as this is
for a group university project, I was hoping i wouldn't have to force
too many workarounds
When you think about it, it is not really hard work around. It is simple and I'm sure your group would be able to apply this workaround easily.
Please follow the below steps.
Create a separate class library in your solution (you can target .NET framework 4).
Then add NUnit assemblies and the NUnit test adapter as usual.
Right click on this project and select 'Add' then 'Existing Item'
Select the Win8 Store Unit Test project and locate the Unit Test file you want to add. When you add the file, make sure you select 'Add As a Link' button. Please see below.
Now rebuild the solution, close and re-open the UnitTest explorer and you should be able to run those tests.
*The reason I said can't guaranteed. I haven't really written Unit tests against Win8 App. So if your SUT (System Under Test) require special configuration it might cause issues. But I'm not sure.
Finally creating a link files are not that hard if everything works you can continue to do this until NUnit has the support for Win8 Unit Testing. Or the other option is simply change all your Unit Tests to use MSTest framework if possible.
Since running Metro apps headlessly is still a gray area: Running a metro app headlessly, I've recently decided to add a native unit test project to my Windows Metro app in hopes that I can find a way to run these unit tests in an automated fashion on the build server. Basically, I'm looking for something similar to MSTest.exe - a utility which is great for running tests from batch files and/or scripts.
In fact, I've tried using the new version of MSTest.exe that comes with VS11 on a generated test .dll, but it fails with the error:
"Unable to load the test container 'test.dll' or one of its dependencies... Error details: Could not load file or assembly file://test.dll' or one of its dependencies. The Module was expected to contain an assembly manifest."
Does MSTest.exe work with test containers that contain WinRT code? If not, is there a utility that will do what I want?
Edit: I just found out that MSTest does not support running tests on a Metro style app. Found here: http://msdn.microsoft.com/en-us/library/ms253138%28v=vs.110%29.aspx This really is too bad. I'm still hoping there's a utility out there that will work.
After blindly digging through the VS folders, I happened to find a new test runner under:
C:\Program Files\Microsoft Visual Studio
11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
This utility allows you to execute WinRT unit tests from the command line. It's very similar to MSTest.exe.
There doesn't seem to be any documentation out there for this yet, but at least a help command exists.
If you are executing vstest.console.exe <filename>.dll then your tests do not get executed in appcontainer mode. You need to give <filename>.appx to execute tests in appcontainer mode. More info on how to execute tests for Windows Metro style apps from command line can be found at Running Unit Tests for Windows Metro style apps from Command Line
There is a documentation about (among other things) running unit tests for Windows Phone:
http://msdn.microsoft.com/en-us/library/windowsphone/develop/dn168930(v=vs.105).aspx
It describes also the command line way of doing it using vstest.console.exe.
It also gives a comparison of supported features between unit tests for Windows Phone and WinRT.
One important detail is that unit tests for WinRT cannot be run on a device. This is a pity and relevant to question How to automate non-interactive tests on Microsoft Surface
I'm trying to work out how to run xUnit tests from TFS 2010.
I found some articles on how to achieve that with the older version (http://jonnekats.wordpress.com/2009/05/07/integrate-xunit-tests-into-your-daily-team-build/, http://weblogs.asp.net/mehfuzh/archive/2009/08/25/configuring-team-build-using-xunit.aspx).
These approaches don't work anymore with TFS 2010 because builds are no longer MSBuild files but instead are Workflows with different tasks.
What I would like to achieve is something similar to what I've got on my dev machine:
- build everything
- run the .xunit project file
- check the results
All that I can see on the web are custom build steps which I can't/won't use because I'll have to configure them for each single unit test assembly and they'll probably get completely messed up in TFS.
Any input would be appreciated.
Why dont use "InvokeProcess" in TFS Build workflow and invoke xUnit command line interface with the necessary xUnit project files as parameters?
You can capture the output of the command line by setting InvokeProcess->stdout to write a build message.
Take a look at the xUnit work item Support VSTS Test Runner on codeplex. You can find there a VS2010 extension in Beta version.