TeamCity VSTest 2015 - unit-testing

I'm trying to setup TeamCity 9.1.7 on a server and I'm encountering an issue when trying to add a Testing step.
Runner Type: Visual Studio Tests
Test engine type: VSTest
Test engine version: VSTest 2015
Test file names: C2.Tests\bin\Debug\C2.Tests.dll
Target platform: x86
When I added this Step, I was prompted to add a Config parameter:
Name: teamcity.dotnet.vstest.14.0
Value: C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe
When I run all my steps I get an error in the Unit Testing step:
[14:23:17][Step 3/3] VSTest report watcher [14:23:17][VSTest report
watcher] No reports found for paths: [14:23:17][VSTest report watcher]
C:\BuildAgent\work\d28aa71801c772cb\TestResults*.trx [14:23:17][Step
3/3] Step Unit Testing (Visual Studio Tests) failed
I've had to do several things differently on this setup (on server) then when I was testing locally on my machine. For example setting up the 2015 Build Tools. I'm not sure where to look to correct this issue.
Please advise.

From the documentation:
The Visual Studio Tests runner integrates MSTest runner and VSTest console runner. Support for both frameworks enables TeamCity to execute tests and automatically import their test results.
The Visual Studio Test Runner requires Visual Studio Test Agent or Microsoft Visual Studio installed on the build agent.
https://confluence.jetbrains.com/display/TCD10/Visual+Studio+Tests
So you can use VSTests perfectly fine from version 10.
For VSTests, you need to specify a different directory for the config parameter:
teamcity.dotnet.vstest.14.0: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
This file will be available after installing the test agent (or visual studio).
If you still get the error then it means there is something wrong with the tests you have specified, you probably didn't reference to the assemblies correctly, make sure the path and .dll files exist.

There are two problems in the current configuration:
You are using as TestEngine the VSTest that will behave differently than the MsBuild
When you run the Test using the MSBuild this will not generate the .trx files that the Report watcher needs to display the execution results.
So in order to fix your problem you need to change the Test Engine to [MSTest]
This link may help you understand the capabilities of each Test Engine
Choose and configure a test runner

If this helps anyone, in my case this exact error was generated when I wrongly identified the DLL file containing the tests. Instead of "IntegrationTests.dll" I wrote "ItegrationTests.dll". You can identify this issue by looking into the build log where team city says:
Command line params:
and then lists paths to all test DLL files. If it's empty, it means the files you have specified in the build step have not been found.
If you're using VS Test runner, you'll see these lines telling you what happened:
[Step 3/3] No test source files were specified.
[Step 3/3] Process exited with code 1
[Step 3/3] VSTest execution failure
If you're using MS Test runner, you'll see these lines instead:
[Step 3/3] Please specify tests to run, or specify the /publish switch to publish results.
[Step 3/3] For switch syntax, type "MSTest /help"
[Step 3/3] Process exited with code 1
[Step 3/3] MSTest execution failure

Try building the tests using the MSBuild step before the testing step - it seems if it can't find the DLLs containing your tests you get the "No reports found" error

Related

Automated unit tests build and run on commandline with Visual Studio solution

I'm working on a project with multiple Unit Tests. I have a visual studio .sln file with around 10 XXPrj in it. Those projects are made with Google Test. Everything works well if I want to run them using Visual Studio 2019, I can build and run the unit tests.
I would like to know what is the best way to run them an automated way with commandline. Purpose is to then integrate this commandline stuff in a jenkins to have everything automated.
Build
Building a Visual Studio solution/project through the command line is done with msbuild.exe. It works best to add the path of MSBuild to the PATH environment variable.
MSBuild is usually installed somewhere in the Visual Studio folders. E.g. on my machine the path is as follows:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MSBuild.exe
Build the solution containing all your projects as follows:
msbuild.exe Example.sln
# Or if you want to build a release version or add additional arguments
msbuild.exe Example.sln /property:Configuration=Debug
See MSBuild CLI Docs for more options.
Side Note: Jenkins has a msbuild plugin you can use with a build step called "Build a Visual Studio project or solution using MSBuild" (Important: this does not install MSBuild, it only provides a GUI to use MSBuild in a build plan).
Run Tests
To run the tests you have two options:
Run each project's executable in your build pipeline and the executable's exit code will indicate the success/failure of the unit tests for that project. However, you will need to call each executable separately; or
Use the vstest.console.exe in combination with the Google Test Adapter
You can use the Google Test Adapter the same way in which Visual Studio uses it when you click Test -> Run -> All tests to discover & execute tests in your projects.
In my environment, vstest.console.exe is located here:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe
You also need to provide the path to the test adaptor. Then execute all the tests as follows:
# Assuming vstext.console.exe is included in the PATH
# and the current working directory is the relevant project executable
# output folder:
vstest.console.exe Project1.exe Project2.exe Project3.exe /TestAdapterPath:"<path to adapter>"
Once again the path is hidden somewhere in the Visual Studio Folders. I found it through searching for GoogleTestAdapter.TestAdapter.dll. On my machine it is located in:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\drknwe51.xnq
Conclusion
Thus, an automated way to build and run GoogleTest unit tests, which are split over multiple projects, with the commandline can be performed in these two steps:
Build the solution/projects using msbuild.exe
Run the tests using vtest.console.exe in combination with the Google Test Adapter

Data collector 'Code Coverage' failed to provide initialization information

I'm trying to create a code coverage report using the Build pipeline.
I have added the task of typeVisual Studio code in the build pipeline and have enabled the Code Coverage.
When the build is triggered. I'm getting :
Data collector 'Code Coverage' message: Data collector 'Code Coverage' failed to provide initialization information. Error: System.TypeInitializationException: The type initializer for 'Microsoft.VisualStudio.Diagnostics.Logging.ProfilerInterop' threw an exception. ---> Microsoft.VisualStudio.Diagnostics.Common.InvariantException: Failed to load IntelliTrace Profiler binary or failed to locate functions.
and
---> System.ComponentModel.Win32Exception: The system cannot find the path specified
This is running the tests and all the tests are passed. However I'm not able to view the code coverage report. The report which it has created contains only information about the tests
Any input on where we specify the path will be useful.
You basically need Visual Studio Test Agent for Code Coverage.
There are 2 possible ways to install for this:
Option 1
Install Agents for Visual Studio 2019 on the Build server (download from here, see under Tools for Visual Studio 2019).
In the build pipeline edit the Visual Studio Test Assemblies task. Set Select test platform using to Specific location and set Path to vstest.console.exe to for example C:\Program Files (x86)\Microsoft Visual Studio\2019\TestAgent\Common7\IDE\Extensions\TestPlatform\vstest.console.exe.
Option 2
Add the Visual Studio test platform installer build task to your pipeline. (Add this task before the testing task).
In the Visual Studio Test Assemblies task you have to select Installed by Tools Installer as the Test platform version
Note: in my experience I had some tests who succeeded with option 1, but failed in option2. Sadly I don't have the time to figure out why...
I faced this issue when configuring a build container.
1 To install the Visual Studio 2019 Test Agent, I used Chocolatey
1.2 Install Chocolatey
ENV chocolateyUseWindowsCompression = false
SHELL ["powershell.exe", "-ExecutionPolicy", "Bypass", "-Command"]
RUN Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1')); `
[System.Environment]::SetEnvironmentVariable('PATH', "\"${env:PATH};%ALLUSERSPROFILE%\chocolatey\bin\"", 'Machine'); `
choco feature enable -n allowGlobalConfirmation;
1.2 Install the TestAgent with chocolatey
RUN choco install visualstudio2019testagent -y
2. Edit the VSTest task on your pipeline to use a specific location.
In my case, I've installed into the container the VSBuildTools. The VSTest task used the VSBuildTools vstest.console.exe but the execution needs some libraries that are located at the TestAgent folders.
vstestLocationMethod: location
vstestLocation: 'C:\Program Files (x86)\Microsoft Visual Studio\2019\TestAgent\Common7\IDE\Extensions\TestPlatform\vstest.console.exe'
Had a similar problem after updating build agent and adding VS2017 capabilities.
Downloaded (from https://www.opendll.com): microsoft.intellitrace.profiler.dll
Add to folder C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\Extensions\TestPlatform\x64​
Good luck

Error while executing the coded UI automated test cases using MTM 17

Test Logs:
"Queued the TMI run for test run [1594]." TestOutcome 'Warning';
Message 'Warning: Test Run deployment issue: The assembly or module
'Microsoft.VisualStudio.Shell.15.0' directly or indirectly referenced
by deployment item 'C:\Program Files (x86)\Microsoft Visual Studio'
specified by the test settings was not found.'."
Several other similar errors.
Not sure why is it showing so.
It sounds like you need to install the appropriate version of the Agents for Visual Studio package on the machine where you're trying to run these tests.
The 2017 version should be what you're after if you're using Visual Studio 2017.

Can't get XUnit tests working with Visual Studio 2017 RC

For the life of me I can't get unit testing working in Visual Studio 2017 from the new msbuild-based netcoreapp1.0 xunit project template.
The requirement is for unit tests to work both inside Visual Studio (for the devs) and from dotnet test on the CLI for the automated build process however, I can't get either working consistently.
Here is what I have tried:
In an existing solution, create a new project and select .NET Core > xUnit Test Project.
Build project from Visual Studio, default test appears and runs successfully, now run dotnet test from powershell prompt, get:
> dotnet test
Test run for D:\...\bin\Debug\netcoreapp1.0\MyProj.dll(.NETCoreApp,Version=v1.0)
dotnet exec needs a managed .dll or .exe extension. The application specified was 'C:\Program'
Or dotnet test with csproj file:
> dotnet test MyProject.csproj
(same error as above)
> dotnet test ..\MySolution.sln
Couldn't find a project to run test from. Ensure a project exists in D:\...
Or pass the path to the project
If I add the xunit.runner.console or xunit.runner.msbuild nuget packages, it stops the unit tests working from inside Visual Studio.
How do I get both working at the same time?
Thanks!
The bug you're hitting is present in Preview 3 and fixed in Preview 4. They didn't escape the command line when executing it, and since dotnet.exe is installed into C:\Program Files\dotnet by default, it always fails.
If you want to continue to use Preview 3, the simplest work-around is to edit your system PATH environment variable, and replace C:\Program Files\dotnet with C:\Progra~1\dotnet.
I know this isn't a very good answer, but dotnet-test-xunit only support project.json files. VS 2017 forces you to switch to csproj files.
I found this on xunit twitter feed:
If you're trying use #xunit in VS2017 RC w/ .NET Core, remove dotnet-test-xunit and use xunit.runner.visualstudio 2.2 beta 4 instead.
With the latest RC.3 I was having issues with the tests not being discovered, and found out that when you run the built-in Test Explorer it says in the output that Microsoft.DotNet.InternalAbstractions 1.0.0 is missing. This was also issue in the previous versions of .NET Core, and the solution is the same, install the package from Nuget.

Automated tests on TFS 2010 using a test assembly file specification

I'm using TFS, it works fine. Now I want to add tests and code coverage. So I create a Local.testsettings. Running tests on my computer runs fine.
Configuration of the TFS Build is as follows:
- Automated test with test assembly file specification and the same test settings file (as recommended, instead of using a .vsmdi file)
When launching a build with TFS, my tests don't get executed :
Run MSTest for Test Assemblies
The MSTestActivity was invoked without a value for Platform or Flavor. The values Mixed Platforms and Debug were used.
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe /nologo /usestderr /testSettings:"C:\Builds\1\MyProject\Sources\MyProject\Main\Source\MyProject\Local.testsettings" /searchpathroot:"C:\Builds\1\MyProject\Binaries" /resultsfileroot:"C:\Builds\1\MyProject\TestResults" /testcontainer:"C:\Builds\1\MyProject\Binaries\MyProject.Tests.Module1.dll" /maxpriority:"1" /minpriority:"1" /publish:"http://tfsserver:8080/tfs/Test" /publishbuild:"vstfs:///Build/Build/433" /teamproject:"MyProject" /platform:"Mixed Platforms" /flavor:"Debug"
Loading C:\Builds\1\MyProject\Sources\MyProject\Main\Source\MyProject\Local.testsettings...
Loading C:\Builds\1\MyProject\Binaries\MyProject.Tests.Module1.dll...
Starting execution...
No tests to execute.
There are no results to be published.
So it finds the assembly, but MSTest doesn't actually runs the tests.
Any hint is welcome.
Looking at this:
/maxpriority:"1" /minpriority:"1"
It looks like your priority criteria may be the problem.