On a project I am working on, I am maintaining some Feature Tests written in SpecFlow. Our team started using Visual Studio 2017 about a year ago, and we finally got around to doing some upkeep on our tests!
Our tests for the project I'm working on were originally written in SpecFlow 2.3.2, and were last updated in Visual Studio 2015.
The SpecFlowSingleFileGenerator is known to not work on VS 2017, so I spent the better part of yesterday changing our suite to use the MSBuildSingleFileGenerator instead as detailed in this article in SpecFlow's official documentation
Problem:
Locally, I can build my solution, including the Feature Test project just fine.
However, I keep getting the following error when I try to build the project on our build server:
[exec] C:\CheckoutDirectory\My Awesome Project\packages\SpecFlow.Tools.MsBuild.Generation.2.3.2\build\SpecFlow.Tools.MsBuild.Generation.targets(45,5):
error MSB4036: The "GenerateAll" task was not found.
Check the following:
1.) The name of the task in the project file is the same as the name of the task class.
2.) The task class is "public" and implements the Microsoft.Build.Framework.ITask interface.
3.) The task is correctly declared with <UsingTask> in the project file, or in the *.tasks files located in the "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin" directory. [C:\CheckoutDirectory\My Awesome Project\AwesomeProject.FeatureTest\AwesomeProject.FeatureTest.csproj]
I should point out that our team has no experience with writing MS Build tasks, as hitherto we haven't needed to; we use NAnt build scripts on TeamCity to manage our build work. It's clear that error message would be helpful...if we knew literally anything about it.
Now, normally the correct answer would be: Google it. I did that, and this specific error has no pertinent results.
Additionally, this is blocking my team, since we need our build to work. I don't have the time to do the research and education necessary to properly understand how MS Build technology works. That wil have to come later.
Question:
Bearing in mind that SpecFlow has broken our process, and our team's lack of knowledge about the MS Build system: I need to know how to get around the "GenerateAll" task was not found error. What do I do to get around it?
Secondary Question:
I'm also open to lateral thinking. Is there some way to hack either VS 2017 or SpecFlow to make the SpecFlowSingleFileGenerator "compatible" with each other? The objective here is NOT to avoid making changes, but to control the changes. I need a path towards transitioning from the old file generator to the MS build generation system.
Additional Information:
So, I did some digging, and I found a place where "GenerateAll" is being called in the SpecFlow.Tools.MsBuild.Generation.targets file:
<Target Name="UpdateFeatureFilesInProject"
DependsOnTargets="BeforeUpdateFeatureFilesInProject"
Inputs="#(SpecFlowFeatureFiles)" Outputs="#(SpecFlowFeatureFiles->'%(RelativeDir)\%(Filename).feature.cs')">
<GenerateAll
ShowTrace="$(ShowTrace)"
BuildServerMode="$(BuildServerMode)"
OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)"
ProjectPath="$(MSBuildProjectFullPath)"
ForceGeneration="$(ForceGeneration)"
VerboseOutput="$(VerboseOutput)"
DebugTask="$(SpecFlow_DebugMSBuildTask)"
>
<Output TaskParameter="GeneratedFiles" ItemName="SpecFlowGeneratedFiles" />
</GenerateAll>
</Target>
Because I've confirmed that this is being copied out to the build server, the situation is yet more mysterious. It appears that the NuGet package is being pulled down faithfully. Therefore, I can't figure out why my local copy is behaving differently than the copy on the build server.
I am not sure where you found this statement:
The SpecFlowSingleFileGenerator is known to not work on VS 2017
The SpecFlowSingleFileGenerator is working in VS2015, VS2017 and VS2019. We see it as a legacy feature, but it's still there. Since some weeks it is disabled by default, but you can enable it in the options.
It works for SpecFlow >= 2.3.2 and 2.4. For SpecFlow 3 you have to use the MSBuild integration. There are some problems with older versions of SpecFlow, but with them it can also work. It depends on your setup.
About your MSBuild error:
The MSBuild Task for SpecFlow < 3.0 is in the specflow.exe. Is it on your build server?
It is part of the SpecFlow NuGet packages. Normally you get this kind of error if MSBuild can't find the assembly where the task is.
For "debugging" problems with MSBuild, I can highly recommend to use the MSBuild Structured Log Viewer (http://msbuildlog.com/). With it, it makes it easy so see what is happening in your build.
We have an example for MSBuild Code- Behind- Generation with SpecFlow 2.3.2 here: https://github.com/techtalk/SpecFlow-Examples/tree/master/MSBuild/OldCSProj_SpecFlow232
You could compare your project with this example.
Full disclosure: I am one of the maintainers of SpecFlow.
I upgraded to Visual Studio 2017 last week. A solution with several test projects using MSTest v1 seemed to be fine at first. Most test projects execute tests fine. This is .NET Framework, not .NET Core.
However, there is one project where I am encountering issues. It's an integration test project, meaning it runs web application tests using the ChromeDriver and Selenium. This project references NUnit, though that reference comes from another project reference. I am not actually using NUnit as my test framework.
This is the error when trying to run any test in that particular test project:
Could not find test executor with URI 'executor://mstestadapter/v1'. Make sure that the test executor is installed and supports .net runtime version .
Test project {name} does not reference any .NET NuGet adapter. Test discovery or execution might not work for this project.
It's recommended to reference NuGet test adapters in each test project in the solution.
Disabling the option indicated below made everything work:
I found this solution after browsing the preview release notes at https://learn.microsoft.com/en-us/visualstudio/releasenotes/vs2017-preview-relnotes#--visual-studio-2017-version-158-preview-4 (scroll down to "... What's New ..." and then .NET Test Adapter Extension). It mentions this breaking change (though it didn't break my other projects), but there is no mention of what to actually do to solve the problem.
Of course, long-term, keeping that option unchecked doesn't seem feasible. So I am looking for guidance on how to proceed.
I already tried to see if I could "quickly" upgrade to MSTest v2, but that had an issue because System.Runtime 4.1.2 couldn't be found. I tried adding System.Runtime 4.3.0 via NuGet, but that didn't solve the issue.
I was getting a similar issue running Xamarin.UITests and resolved it by installing the "NUnit 2 Test Adapter" Extension in Visual Studio. In Visual Studio 2017 you do this from Tools > > Extensions and Updates... > Online > Search "NUnit Adapter" > click the NUnit 2 Test Adapter then the Download button to install. Restart Visual Studio when prompted to complete the installation.
It would be interesting to understand how the MSTest V1 framework assembly(Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll) is referenced in this project. The compatibility checkbox that also ended up being disabled in the image above should ensure that this project has MSTestV1 tests and enabled run tests. If it isn't then for some reason the detection logic is busted and needs to be fixed - a snippet on how the assembly is referenced would be helpful there.
As for moving to MSTest V2, this blog should help: https://blogs.msdn.microsoft.com/devops/2016/06/17/taking-the-mstest-framework-forward-with-mstest-v2/ . And for the specific issue you seem to be running into, this github issue on the repo sounds similar: https://github.com/Microsoft/testfx/issues/184 .
Caltor's answer pretty much solved it for me: it was the NUnit Adapter I needed. But I followed slightly different steps. Here they are for VS 2019 (but probably apply to VS 17 too):
Right clicked on the References node under my project in the solution explorer
Chose "Manager NuGet packages"
Clicked Browse
Searched for nunit3testadapter
Hit install on the result that popped up
The issue for me was similar to the OP's - the adapter was installed on some projects but not others. But confusingly this wasn't immediately visible under the "References" node for that project. The only node relating to nunit under that References node said "nunit.framework". So the references for test with and without the adapter looked the same.
"... but that had an issue because System.Runtime 4.1.2 couldn't be found."
I think this is a setting in the app.config that does not get updated when the project settings are updated. I searched for System.Runtime under the packages folder and then changed my app.config to match what I found, and then that particular problem went away.
I'm working on a Django (1.8.6) project and using Visual Studio Online's GIT source control. I am building the application successfully in my local environment and push the changes to the VSO. However, whenever I try to build the application on VSO to be able to benefit from "Continuous Integration" as a next step (will try to deploy Azure), it fails by giving the error below:
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Python
Tools\Microsoft.PythonTools.Web.targets (235, 5) The environment 'env
(Python 3.4) (unavailable)' is not available. Check your project
configuration and try again.
Unexpected exit code received from msbuild.exe: 1
My build definition on VSO:
Build Definition Screenshot
Python Version: 3.4.3
VS Version: 2015
Any suggestions regarding to my case is highly appreciated.
This is a known issue for PTVS. MS is still working on it. Refer to this thread for details: https://github.com/Microsoft/PTVS/issues/128
There is a workaround in that thread you may use, I quote it here. The second link is unavailabe now, but the first one still works.
For deployment via PowerShell, I found this, which looks correct
AFAIK.
http://www.kenneth-truyers.net/2014/02/06/deploying-cloud-services-to-azure-with-powershell/
You can also use the Python Azure SDK to deploy, but that's not as
well documented. This is what this test does:
https://github.com/Azure/azure-sdk-for-python/blob/master/tests/test_servicemanagementservice.py#L940
Both of these assume that you are able to create the .cspkg, as that's
the file you have to upload to blob storage.
EDIT 2016-10-19:
The original question was about an issue specific to VS2015 CTP6 with the XUnit test runner. It's clear from the answers that there is a much broader issue with unit test discovery in Visual Studio which may occur in many different situations. I have cleaned up my question to reflect that.
I have also included a script in my own answer that I still use to this day to solve similar problems when they appear.
Many other answers have also proven helpful in better understanding the intricacies of the VS test runner. I appreciate that people are still sharing their solutions!
Original question 2015-04-10:
Since yesterday, my Visual Studio Test Explorer won't discover tests for any of my projects. It does not show the green loading bar after building, either.
When I go to the Visual Studio Test Explorer and click "Run All", or when I right-click any test method and select "Run Tests", I get the following in my output window:
Could not load file or assembly 'Microsoft.VisualStudio.Web.ProjectSystem, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
I am running Visual Studio 2015 CTP 6 on Windows 10 Pro Technical Preview, build 10041. The .NET Framework version does not seem to matter - it happens on 4.0, 4.5.2 and 4.6.
I tried with the following testing frameworks and all of them give the same behavior:
Microsoft.VisualStudio.QualityTools.UnitTestFramework v14.0.22609.0
xunit v2.1.0-beta1-build2945 with xunit.runner.visualstudio v2.1.0-beta1-build1051
NUnit v2.6.4 with NUnitTestAdapter v2.0.0
I found an issue on GitHub (xunit) that appeared to be similar: Cannot get tests discovered #295, with this comment from the xunit team:
Be aware that Visual Studio 2015 CTP 5 has been reported to be broken
by many people with unit testing in general (not just xUnit.net), so
don't expect that to work.
Also, please make sure you've cleaned out Visual Studio's runner
cache. If it gets corrupted, Visual Studio will permanently misbehave
until it's deleted. To clear the cache, shut down all instances of
Visual Studio, then delete the folder
%TEMP%\VisualStudioTestExplorerExtensions (honestly, it probably
wouldn't hurt to delete everything in %TEMP% that can be deleted).
I tried their suggestion to delete the folder %TEMP%\VisualStudioTestExplorerExtensions. Unfortunately that did not fix the problem.
I noticed that ReSharper actually is able to discover some tests. It only works for the VS and NUnit tests, not for xunit.
There has to be some sort of temp or cache folder I need to clear, but I know Visual Studio has many of them and not all of them can be deleted without unwanted side-effects.
To my surprise, clearing temp files located in the %TEMP% directory resolved the issue for me.
Note: This path is generally at C:\Users\(yourusername)\AppData\Local\Temp
As #Warren-P included, you can navigate to the temp folder by putting in %temp% in Start Menu, or launch "File Explorer" and enter %temp% in the address bar.
It could be that your code is compiled with x64 therefore have to enable the Default Processor Architecture as X64.
Test > Test Settings > Default Processor Architecture > X64
Check out, if NUnit Test Adapter 2/3 is installed in VisualStudio.
(Tools>Extensions and Updates )
Make sure that correct processor architecture is chosen:
(Test>Test Settings>Default Processor Architecture)
EDIT 2016-10-19 (PowerShell script)
This issue still returns every now and then. I wrote a small PowerShell snippet to automate clearing the relevant cache/temp folder/files for me. I'm sharing it here for future readers:
#(
"$env:TEMP"
"$env:LOCALAPPDATA\Microsoft\UnitTest"
"$env:LOCALAPPDATA\Microsoft\VisualStudio\14.0\1033\SpecificFolderCache.xml"
"$env:LOCALAPPDATA\Microsoft\VisualStudio\14.0\1033\ProjectTemplateMRU.xml"
"$env:LOCALAPPDATA\Microsoft\VisualStudio\14.0\ComponentModelCache"
"$env:LOCALAPPDATA\Microsoft\VisualStudio\14.0\Designer\ShadowCache"
"$env:LOCALAPPDATA\Microsoft\VisualStudio\14.0\ImageLibrary\cache"
"$env:LOCALAPPDATA\Microsoft\VisualStudio Services\6.0\Cache"
"$env:LOCALAPPDATA\Microsoft\WebsiteCache"
"$env:LOCALAPPDATA\NuGet\Cache"
) |% { Remove-Item -Path $_ -Recurse -Force }
Make sure to close Visual Studio beforehand and it's probably a good idea to reboot afterwards.
Deleting the TEMP folder may not be necessary and may in some cases even be undesirable, so I would recommend trying without clearing the TEMP folder first. Just omit the "$env:TEMP".
Original answer 2015-04-12
The problem was "solved" after a thorough cleaning of Visual Studio-related temp/cache folders.
Since I did not have the time to go through everything one-by-one and then test in-between, I unfortunately don't know which one actually caused the problem.
These are the exact steps I've taken:
Closed Visual Studio
Used CCleaner to clear system and browser temp files/folders
Manually cleared/deleted the following files/folders:
%USERPROFILE%\AppData\Local\assembly
%USERPROFILE%\AppData\Local\Microsoft\UnitTest
%USERPROFILE%\AppData\Local\Microsoft\VisualStudio\14.0\1033\SpecificFolderCache.xml
%USERPROFILE%\AppData\Local\Microsoft\VisualStudio\14.0\1033\ProjectTemplateMRU.xml
%USERPROFILE%\AppData\Local\Microsoft\VisualStudio\14.0\ComponentModelCache
%USERPROFILE%\AppData\Local\Microsoft\VisualStudio\14.0\Designer\ShadowCache
%USERPROFILE%\AppData\Local\Microsoft\VisualStudio\14.0\ImageLibrary\cache
%USERPROFILE%\AppData\Local\Microsoft\VisualStudio Services\6.0\Cache
%USERPROFILE%\AppData\Local\Microsoft\WebsiteCache
%USERPROFILE%\AppData\Local\NuGet\Cache
%USERPROFILE%\AppData\Local\Temp
One reason for this problem is that your test class is not public. MSTest only discovers tests from public classes.
In Visual Studio 2015(Update 3) if you want to attach the tests in the test explorer then have to install the NUnit Test Adapter.Download the adapter from Tools->Extension And Updates->Online tab(you have to search for the adapter)->Download.
By restarting the Visual Studio you can see the change for the test framework.
I don't have a complete answer to this, but I have determined some things by playing with a test project:
The xunit.runner.aspnet : 2.0.0-aspnet-beta4 that appears to be part of the official beta4 aspnet5 release does not work in Visual Studio.
Instead, using "xunit": "2.1.0-*" and "xunit-runner.dnx": "2.1.0-*" packages DO work in Visual Studio.
In order for VS to discover the tests, your project MUST have a SINGLE command named "test" that runs "xunit.runner.dnx". Adding additional commands may break it.
If your Test Explorer window still ends up empty, REMOVE the "test" command from your project, then re-build the solution, then add the "test" command back to the project.json.
Clearing all your caches as per #Fred-Kleuver 's suggestion may help, but I haven't done all the steps in isolation, so I'm not sure.
This is current as per VS 2015 CTP 6, using the beta4 releases, not the dailies.
I had an instance where some tests would not be picked up because I had made them async like the following:
public async void This_IsMy_UnitTest()
The problem was I forgot to make them return a Task and not void when I did the switch-over. One would think this would cause an error or failed test but nope. The unit tests in that class were fully ignored and acted like they didn't exist.
It wasn't after about 3 clean and builds + restarting VS.NET that I saw the test run and fail indicating I forgot to add the Task return type:
public async Task This_IsMy_UnitTest()
After the update, the unit tests were found and worked correctly. This might be an edge case, but having async tests for using await within but not having the signature correct can cause this same issue and it's not the 1st time I've done this.
Go to Nuget package manager and download Nunit Adapter as follow.
I had the same pronlem but the folder "%TEMP%\VisualStudioTestExplorerExtensions" did not exist on my machine so as i read the posts i had the idea to create it and it works. The test explorer is now able to show all my tests.
Thanks.
Just restart Visual Studio and in Test Explorer do "Run All" ... All my tests are discovered then.
In my case (Visual Studio Enterprise 2015 14.0.25425.01 Update 3, Resharper 2016.2) I just needed to do a clean solution from the Build menu. Rebuilding the solution then cause the test explorer to "wake up" and find all the tests again.
The solution in my case was just to install the NUnit 3 Test Adapter extension to my Visual Studio 2015.
In my case, the problem was "between the chair and keyboard". I had switched to a configuration in the Configuration Manager which did not include my unit test projects on build. Switching back to a configuration (e.g. Debug) which includes all projects fixed the issue.
In my case, MSTest under VS 2015 was ignoring tests with test (i.e. method) names that were longer than 174 characters. Shortening the name allowed the test to be visible. This was determined via guess-and-check by manipulating the test name.
This probably won't help most people, but someone inexperienced at unit testing had written a test method that returned bool instead of void:
[TestMethod]
public bool TestSomething()
Changing the return type to void fixed the problem.
Ensure you have xunit.runner.visualstudio package in your test project packages.config and also that was correctly restored.
I know this was not the case of the original question however it could save time for someone like me.
I would just like to add that I found an entirely different solution to the ones above.
I had declared my test class as below:
[TestClass]
class ClassificationTests
{
//unit tests
}
As soon as I added the public modifier to the class, it worked as expected!
This topic is somewhat outdated, but my solution to the missing Test status in VS2015:
The task-status only shows up on the Debug buildconfiguration. Ofcourse this also makes it impossible to debug your test through the test-explorer.
Somehow my project was set to compile as a Static Library (.lib). After changing this to a Dynamic Library (.dll), tests where discovered correctly by Visual Studio 2012.
My Unit Test Project ->
Properties ->
Configuration Properties ->
General ->
Configuration Type
If you're targetting .NET Standard or .NET Core, you need to use the NuGet package for NUnit Test Adapter and not the extension.
It is recommended to install the adapter from NuGet if you are testing .NET Core or .NET Standard projects. The VSIX adapter does not, and will not, support .NET Core because VSIX packages cannot target multiple platforms.
Source: NUnit GitHub Wiki
.
Also check the FAQ there:
My tests aren't showing up in Visual Studio 2017?
Are you using the NuGet package?
Are you using version 3.8.0 or newer of the NuGet package?
Do your tests target .NET Core or the full .NET Framework? (see above)
Have you added a Package Reference to Microsoft.NET.Test.Sdk?
Have you restarted Visual Studio? It is still a bit tempermental.
Source: NUnit GitHub Wiki
This happened to me because my test project contained an app.config.
It was automatically added by NuGet packages for assembly redirection, but my tests seemed to run fine without it.
See: https://developercommunity.visualstudio.com/comments/42858/view.html.
I had the same problem. I just cleaned and rebuilt the project and I was able to see the tests that were missing.
Popping in to share my solution. I was on Windows 10, Visual Studio 2015, NUnit 3.5, NUnit Test Adapter 3.6 (via NuGet, not the VISX extension) and none of my tests were being discovered. My problem was that in the Tests project of my solution, somehow a shortcut to my "Documents" folder had been created within the project folder. I'm guessing the test adapter was seeing the shortcut and getting hung up trying to figure out what to do with it, resulting in the failure to display unit tests.
Deleting the file \AppData\Local\Microsoft\VisualStudio\14.0\1033\SpecificFolderCache.xml solved the issue for me.
I was also bitten by this wonderful little feature and nothing described here worked for me. It wasn't until I double-checked the build output and noticed that the pertinent projects weren't being built. A visit to configuration manager confirmed my suspicions.
Visual Studio 2015 had happily allowed me to add new projects but decided that it wasn't worth building them. Once I added the projects to the build it started playing nicely.
I resolved it by changing X64 to :
Right click on project -> Properties -> Build -> Platform target -> Any CPU
It was so easy for me to fix the issue as:
Select your Unit Test Project
Click on 'Show All Files' button in the Solution Explorer and new temporary files appeared in the file tree of the Solution Explorer within 'obj\x86\Debug'.
Delete these temporary files and rebuild project.
Retried to run tests and worked!.
We had the same problem. We have a big VS 2015 solution with multiple C# projects in it and even more test projects.
Resharper's test discovery worked just fine, but VS Test Explorer failed miserably.
Turns out that the projects didn't have the same version of MsTest TestFramework and TestAdapter, and that sometimes they used NuGets and other times good old references, and that is not supported apparently (so much for such an expensive IDE).
Removing all Microsoft.VisualStudio.Test* references and then adding / updating the two MSTest NuGets fixed the problem.
I solved this problem by realizing that the Target Framework for my test project was different than the project under test. Yes, I caused this problem by changing the target framework from the default (Project>Properties>Application), but failed to this this for the test project, which was created several weeks later. The mismatch did not cause a compiler error, but it did result in a warning in the Error List window. Once I selected the option to display warnings, the solution was obvious.