vstest.console throws System.IO.FileNotFoundException - unit-testing

My application has several unit tests projects. All other projects and the unit tests projects get built to a common output directory. With an msbuild task I'm collecting all the unit tests assemblies and run vstest.console.exe to test these assemblies.
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\..\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" "Assembly1.UnitTest.dll" "Assembly2.UnitTest.dll" /Platform:x64 /Framework:Framework40 /InIsolation /Logger:trx
This results in an exception: System.IO.FileNotFoundException: Could not load file or assembly 'someassembly.dll' or one of its dependencies. The specified module could not be found.
When I execute vstest.console.exe for a single unit tests assembly the above exception doesn't occur. Further investigation learned me that when testing multiple assemblies at once the vstest.console is copying the test assemblies and depended assemblies to an "out" directory in the "testresults" directory. However not all needed assemblies are referenced by a project but manually copied to the common output directory. Those assemblies are missing in the "out" directory in "testresults" and causing the System.IO.FileNotFoundException.
With MSTest I could use a config file an point to the common output directory by adding a DeploymentItem. For vstest.console this doesn't work anymore.
What can I do to get around this behavior? I don't want to work with an "out" directory. Running my unit tests from the common output directory is just fine.
PS. I have the same issue on TFS 2013 with build definitions. My build definitions are collecting *.unittest.dlls and executing these with the Test Runner.

The solution is using a .runsettings file. In the runsettings file it is possible to specify "DeploymentEnabled". By default this is true. Changing it to false doesn't copy all the assemblies to the out directory of the TestResults directory. More information on the runsettings file: http://msdn.microsoft.com/en-us/library/jj635153.aspx.
Example:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!--Info: http://msdn.microsoft.com/en-us/library/jj635153.aspx -->
<MSTest>
<IgnoreTestImpact>True</IgnoreTestImpact>
<MapInconclusiveToFailed>True</MapInconclusiveToFailed>
<CaptureTraceOutput>False</CaptureTraceOutput>
<DeleteDeploymentDirectoryAfterTestRunIsComplete>True</DeleteDeploymentDirectoryAfterTestRunIsComplete>
<DeploymentEnabled>False</DeploymentEnabled>
</MSTest>
</RunSettings>

Related

Dotnet command failed with non-zero exit code on the following projects

I have set up a build definition on VSTS. It builds fine on my local machine. But I get the following error when I start the build process
[error]Dotnet command failed with non-zero exit code on the following projects : D:\a\1\s\My.Project\My.Project.csproj
Check the project name in the error carefully. Chances are its not actually in your solution .sln ... and wouldn't build if it were. By default, VSTS will try to build all .csproj files regardless whether they are in your solution or not. Remove all .csproj files for projects you don't want to be built.
In my case. I deleted the \My.Project directory (which contained the .csproj file) and the build suceeded.
edit your.csproj and add MvcRazorCompileOnPublish as false
<PropertyGroup>
<TargetFramework>netcoreapp2.0</TargetFramework>
<PreserveCompilationContext>true</PreserveCompilationContext>
<MvcRazorCompileOnPublish>false</MvcRazorCompileOnPublish>
</PropertyGroup>
views will be included in publish without the VSTS build failure.
In my case, my project had the line <IsPackable>false</IsPackable> in the csproj file, which caused the Pipeline job to fail in DevOps when it tried to create a package.
I faced similar issue with my automation code's Build pipeline with CI. I was using ".Net Core" task for test execution.
Solution:
I switched my "test" execution task from .Net Core to "Visual Studio test".
In VSTest task under "Test files" i specified exact project name which contains Test Cases. Instead of main solution.dll. As shown in image.

XAML build not able to run Unit Test

I am trying to run Nunit Test using Visual Studio test Runner from XAML build in TFS,but i am getting error saying
TF900547: The directory containing the assemblies for the Visual Studio Test Runner is not valid ''.
Based on your history, I assume you are using VSTS, you can refer to these steps to run Nunit test:
Create a unit test project via VS
Install NUnit and NUnit3TestAdapter packages
Add solution to source control (if the package files not added to the source control, you can refer to the steps below)
Open Source Control Explorer in VS
Add a new folder (e.g. Tools) and add Nuget.exe to the source control
Add bat file to your test project (For example: Tools\nuget35.exe restore NUTest2\NUTest2.sln)
Open XAML build definition and map Tools source folder to agent
Select Process section
Choose TfvcTemplate.12.xaml build template
In the Advance section of Build, specify Pre-build script path with previous bat file.

VS2012 Unit Tests: How to change the location of the TestResults folder

I have all my Unit Test project in a folder under my Solution Folder and would like to have the TestResults folder in the same folder as the Test projects instead in the solution directory.
I have found that this could be done via the test seeting file:
How to specify the location for the unit test results in VS 2010?
but I also read, that with VS2012 you should no longer use tests settings files. Actually VS2012 does not create one.
Is there another way?
To specify a different location for the "TestSettings" folder, add a .runsettings to your solution as explained in the Visual Studio documentation: http://msdn.microsoft.com/en-us/library/vstudio/jj635153.aspx
My .runsettings file looks like this:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<RunConfiguration>
<ResultsDirectory>.\Visual Studio Test Results</ResultsDirectory>
</RunConfiguration>
</RunSettings>
As far as I could tell, however, the ResultsDirectory location is not relative to the solution folder (like the example file from the doc suggests), but rather relative to the location of the .runsettings file itself. Also note that Visual Studio macros like $(SolutionDir) are not expanded here.
All in all, .runsettings files are not related to a particular project or solution.
The reason why they recommend using .runsettings files instead of .testsettings in newer version of Visual Studio is also found in the documentation: http://msdn.microsoft.com/en-us/library/vstudio/ee256991.aspx
If you use a .testsettings file, the MSTest test framework will be
used to run your tests. This runs more slowly and does not allow you
to run tests from third-party test frameworks.
You can create a small RunSettings file which looks like
<RunSettings>
<RunConfiguration>
<ResultsDirectory>e:\myResultsFolder</ResultsDirectory>
</RunConfiguration>
</RunSettings>
Select this setting file via top level menu Test->TestSettings->"Select Test Settings" before running your tests.
You can find more detail on http://msdn.microsoft.com/en-us/library/jj635153.aspx.

nunit-agent seems to be failing to load probing privatePath from tests configuration

Previously, when I had only Visual Studio 2010, my unit tests were executing fine.
Basically, my tests are composed of two files: UnitTests.dll and UnitTests.dll.config. Where UniTests.dll.config had a custom probing privatePath (e.g., Public;Extensions;Lib)
In order to execute, I used to follow this workflow:
1. I copied both files (i.e. UnitTests.dll and .config) to the folder were my application under test is located.
2. Open NUnit gui.
3. Execute the tests with ShadowCopy disabled, because my tests need to load the dlls from my application under test.
This was working fine!
After I installed Visual Studio 2012, the tests were not running anymore. And later, I figured out a workaround, but it is something that I don't want to use in my solution.
Now, I have to follow this workflow to make the tests running:
I copy both files (i.e. UnitTests.dll and .config) to the folder where my application under test is located.
I copy all the NUnit installation files (i.e. nunit-agent, nunit-console, etc.) to the folder where my application under test is located.
I change the probing privatePath from nunit-agent.dll.config in order to include the same paths from my UnitTests.dll.config.
Open NUnit gui which is located in my application under test folder.
Execute the tests with ShadowCopy disabled.
Note that I had to include steps 2 and 3 in order to run my unit tests. Somehow, I think nunit-agent.dll is not loading the probing privatePath from the config file of my test assembly.
Does anyone know why this is happening? Does anyone have a workaround where I don't need to change the nunit-agent.dll.config and copy the nunit installation files?
Thanks in advance.

Unit tests aren't run with multiple build configurations in TFS 2010

I am using MSBuild in my TFS Buildprocess using the DefaultTemplate to build multiple configurations(Debug/Release) of the same solution. When I build only release or only debug the unit tests are run but when I run the build with both configurations the unit tests are not run.
When I am running multiple configurations I have each of them Defined in the build definition under Process-> Items to Build-> Configurations to Build which is Mixed Platforms|Release,Mixed Platforms|Debug. When I run a single configuration I am using Mixed Platforms|Release or Mixed Platforms|Debug.
I am unable to put the entire log here(It is over 6000 lines long) but it appears that it isnt finding any of the testAssemblies.
Edit:
Here are the actual sections of the log. I'm not sure where Mixed Platforms\Release is coming from in the searchpathroot.
I should also add that we compile into our Source's folder because a few of our scripts only work in that layout.
Debug Only(works)
Run MSTest for Test Assemblies
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe /nologo /usestderr /testSettings:"C:\Builds\6\ProductName\BuildName\Sources\Product\Local.testsettings" /searchpathroot:"C:\Builds\6\ProductName\BuildName\Sources" /resultsfileroot:"C:\Builds\6\ProductName\BuildName\TestResults" /testcontainer:"C:\Builds\6\ProductName\BuildName\Sources\Test\UnitTestCommon\bin\Debug\UnitTestCommon.dll" /testcontainer:"C:\Builds\6\ProductName\BuildName\Sources\Test\UnitTestCommon\bin\Release\UnitTestCommon.dll" /testcontainer:"C:\Builds\6\ProductName\BuildName\Sources\Test\UnitTestWCF\bin\Debug\UnitTestWCF.dll" /testcontainer:"C:\Builds\6\ProductName\BuildName\Sources\Test\UnitTestWCF\bin\Release\UnitTestWCF.dll" /publish:"http://dctfs2010.company.dc:8080/tfs/Product" /publishbuild:"vstfs:///Build/Build/964" /teamproject:"ProductName" /platform:"Mixed Platforms" /flavor:"Debug"
Loading C:\Builds\6\ProductName\BuildName\Sources\Product\Local.testsettings...
Loading C:\Builds\6\ProductName\BuildName\Sources\Test\UnitTestCommon\bin\Debug\UnitTestCommon.dll...
Loading C:\Builds\6\ProductName\BuildName\Sources\Test\UnitTestCommon\bin\Release\UnitTestCommon.dll...
Debug and Release(Does not work)
Run MSTest for Test Assemblies
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe /nologo /usestderr /testSettings:"C:\Builds\6\ProductName\BuildName\Sources\Product\Local.testsettings" /searchpathroot:"C:\Builds\6\ProductName\BuildName\Sources\Mixed Platforms\Release" /resultsfileroot:"C:\Builds\6\ProductName\BuildName\TestResults" /testcontainer:"C:\Builds\6\ProductName\BuildName\Sources\Test\UnitTestCommon\bin\Debug\UnitTestCommon.dll" /testcontainer:"C:\Builds\6\ProductName\BuildName\Sources\Test\UnitTestCommon\bin\Release\UnitTestCommon.dll" /testcontainer:"C:\Builds\6\ProductName\BuildName\Sources\Test\UnitTestWCF\bin\Debug\UnitTestWCF.dll" /testcontainer:"C:\Builds\6\ProductName\BuildName\Sources\Test\UnitTestWCF\bin\Release\UnitTestWCF.dll" /publish:"http://dctfs2010.company.dc:8080/tfs/Product" /publishbuild:"vstfs:///Build/Build/962" /teamproject:"ProductName" /platform:"Mixed Platforms" /flavor:"Release"
Directory "C:\Builds\6\ProductName\BuildName\Sources\Mixed Platforms\Release" not found.
If you click on the properties of your solution (the solution that you build as part of your build). You will have configuration manager. At the configuration manager you will be able to see 'Debug', 'Released', 'Mixed', etc any other configurations you may have defined. Can you see the test project checked to be build in the mixed configuration? If this is not being build, the test.dll will not be available which means, MSBuild will not have any tests to run.
Another way to verify why the tests are not running, is to look at the build drop directory and see if you can see the test.dll there. If the test.dll is not there then the problem is with how you have the projects configured in the configuration manager.
Turns out my outputDirectory was setup wrong for how our projects are compiled. Which is where the searchpathroot problem came from.