Automated unit tests build and run on commandline with Visual Studio solution - c++

I'm working on a project with multiple Unit Tests. I have a visual studio .sln file with around 10 XXPrj in it. Those projects are made with Google Test. Everything works well if I want to run them using Visual Studio 2019, I can build and run the unit tests.
I would like to know what is the best way to run them an automated way with commandline. Purpose is to then integrate this commandline stuff in a jenkins to have everything automated.

Build
Building a Visual Studio solution/project through the command line is done with msbuild.exe. It works best to add the path of MSBuild to the PATH environment variable.
MSBuild is usually installed somewhere in the Visual Studio folders. E.g. on my machine the path is as follows:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MSBuild.exe
Build the solution containing all your projects as follows:
msbuild.exe Example.sln
# Or if you want to build a release version or add additional arguments
msbuild.exe Example.sln /property:Configuration=Debug
See MSBuild CLI Docs for more options.
Side Note: Jenkins has a msbuild plugin you can use with a build step called "Build a Visual Studio project or solution using MSBuild" (Important: this does not install MSBuild, it only provides a GUI to use MSBuild in a build plan).
Run Tests
To run the tests you have two options:
Run each project's executable in your build pipeline and the executable's exit code will indicate the success/failure of the unit tests for that project. However, you will need to call each executable separately; or
Use the vstest.console.exe in combination with the Google Test Adapter
You can use the Google Test Adapter the same way in which Visual Studio uses it when you click Test -> Run -> All tests to discover & execute tests in your projects.
In my environment, vstest.console.exe is located here:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe
You also need to provide the path to the test adaptor. Then execute all the tests as follows:
# Assuming vstext.console.exe is included in the PATH
# and the current working directory is the relevant project executable
# output folder:
vstest.console.exe Project1.exe Project2.exe Project3.exe /TestAdapterPath:"<path to adapter>"
Once again the path is hidden somewhere in the Visual Studio Folders. I found it through searching for GoogleTestAdapter.TestAdapter.dll. On my machine it is located in:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\drknwe51.xnq
Conclusion
Thus, an automated way to build and run GoogleTest unit tests, which are split over multiple projects, with the commandline can be performed in these two steps:
Build the solution/projects using msbuild.exe
Run the tests using vtest.console.exe in combination with the Google Test Adapter

Related

Data collector 'Code Coverage' failed to provide initialization information

I'm trying to create a code coverage report using the Build pipeline.
I have added the task of typeVisual Studio code in the build pipeline and have enabled the Code Coverage.
When the build is triggered. I'm getting :
Data collector 'Code Coverage' message: Data collector 'Code Coverage' failed to provide initialization information. Error: System.TypeInitializationException: The type initializer for 'Microsoft.VisualStudio.Diagnostics.Logging.ProfilerInterop' threw an exception. ---> Microsoft.VisualStudio.Diagnostics.Common.InvariantException: Failed to load IntelliTrace Profiler binary or failed to locate functions.
and
---> System.ComponentModel.Win32Exception: The system cannot find the path specified
This is running the tests and all the tests are passed. However I'm not able to view the code coverage report. The report which it has created contains only information about the tests
Any input on where we specify the path will be useful.
You basically need Visual Studio Test Agent for Code Coverage.
There are 2 possible ways to install for this:
Option 1
Install Agents for Visual Studio 2019 on the Build server (download from here, see under Tools for Visual Studio 2019).
In the build pipeline edit the Visual Studio Test Assemblies task. Set Select test platform using to Specific location and set Path to vstest.console.exe to for example C:\Program Files (x86)\Microsoft Visual Studio\2019\TestAgent\Common7\IDE\Extensions\TestPlatform\vstest.console.exe.
Option 2
Add the Visual Studio test platform installer build task to your pipeline. (Add this task before the testing task).
In the Visual Studio Test Assemblies task you have to select Installed by Tools Installer as the Test platform version
Note: in my experience I had some tests who succeeded with option 1, but failed in option2. Sadly I don't have the time to figure out why...
I faced this issue when configuring a build container.
1 To install the Visual Studio 2019 Test Agent, I used Chocolatey
1.2 Install Chocolatey
ENV chocolateyUseWindowsCompression = false
SHELL ["powershell.exe", "-ExecutionPolicy", "Bypass", "-Command"]
RUN Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1')); `
[System.Environment]::SetEnvironmentVariable('PATH', "\"${env:PATH};%ALLUSERSPROFILE%\chocolatey\bin\"", 'Machine'); `
choco feature enable -n allowGlobalConfirmation;
1.2 Install the TestAgent with chocolatey
RUN choco install visualstudio2019testagent -y
2. Edit the VSTest task on your pipeline to use a specific location.
In my case, I've installed into the container the VSBuildTools. The VSTest task used the VSBuildTools vstest.console.exe but the execution needs some libraries that are located at the TestAgent folders.
vstestLocationMethod: location
vstestLocation: 'C:\Program Files (x86)\Microsoft Visual Studio\2019\TestAgent\Common7\IDE\Extensions\TestPlatform\vstest.console.exe'
Had a similar problem after updating build agent and adding VS2017 capabilities.
Downloaded (from https://www.opendll.com): microsoft.intellitrace.profiler.dll
Add to folder C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\Extensions\TestPlatform\x64​
Good luck

cppunittest.h no such file or directory

I try build an unit test C++ project on a CI server. On my PC no problem, but on the CI server, I have the message: "'CppUnitTest.h' : No such file or directory".
On both computer (mine and server), there is VS enterprise 2017, the only thing different it's on my PC, I build on VS, and on the CI server, I use MSBuild by command line.
I tried to include additional directories in the project as:
AdditionalIncludeDirectories: $(VCInstallDir)Auxiliary\VS\UnitTest\include;$(VCInstallDir)UnitTest\include;%(AdditionalIncludeDirectories)
AdditionalLibraryDirectories - $(VCInstallDir)UnitTest\lib;$(VCInstallDir)Auxiliary\VS\UnitTest\lib;%(AdditionalLibraryDirectories)
But it doesn't work.
Is there another directory to include ?
cppunittest.h no such file or directory
Since you have installed the Visual Studio 2017 on your build server and build the project successfully, you should use the MSBuild from the Visual Studio instead of the build tool.
You can try to call the MSBuild from following path:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\MSBuild.exe
I have create a unit test C++ project sample to test, and it works fine, if I call the MSBuild from above folder.
Hope this helps.

The directory containing the assemblies for the Visual Studio Test Runner is not valid

Our CI environment setup with [TFS Server 2015 + Visual studio build tools 2017]. When I try to run C# based unit test on environment it shows this error
TF900547: The directory containing the assemblies for the Visual Studio Test Runner is not valid ''.
Then I tried to fix the error by fallowing ways
Installed VS test agent 2015 in the machine https://learn.microsoft.com/en-us/visualstudio/test/lab-management/install-configure-test-agents
Copied C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow from my computer to exact location to CI machine.
Changed test runner from Visual Studio Test Runner to MS Test Runner
But still same error. Looks like tfs server not able to find the test agent folder. Anyone has idea how to fix this
Seems you have VS 2017 Build Tools installed only, no VS IDE on the CI environment.
You need to install VS IDE. VS build tools doesn't contain the "test
platform". In order to run the tests, you require to install full
IDE.
Currently to run the VsTest task, the task has dependency on Visual Studio IDE. But some changes will have in the near future.
We have decided to go with a 'tools installer' which can be used to
get the test platform instead of needing the IDE. If you have the IDE,
great. If you don't, the tools installer will get the test platform on
the agent for you. It is already on backlog and expected to be available on VSTS in a few sprints.
More details please take a look at the discussion in this issue: vstest capability not detected when installed without Visual Studio

TeamCity VSTest 2015

I'm trying to setup TeamCity 9.1.7 on a server and I'm encountering an issue when trying to add a Testing step.
Runner Type: Visual Studio Tests
Test engine type: VSTest
Test engine version: VSTest 2015
Test file names: C2.Tests\bin\Debug\C2.Tests.dll
Target platform: x86
When I added this Step, I was prompted to add a Config parameter:
Name: teamcity.dotnet.vstest.14.0
Value: C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe
When I run all my steps I get an error in the Unit Testing step:
[14:23:17][Step 3/3] VSTest report watcher [14:23:17][VSTest report
watcher] No reports found for paths: [14:23:17][VSTest report watcher]
C:\BuildAgent\work\d28aa71801c772cb\TestResults*.trx [14:23:17][Step
3/3] Step Unit Testing (Visual Studio Tests) failed
I've had to do several things differently on this setup (on server) then when I was testing locally on my machine. For example setting up the 2015 Build Tools. I'm not sure where to look to correct this issue.
Please advise.
From the documentation:
The Visual Studio Tests runner integrates MSTest runner and VSTest console runner. Support for both frameworks enables TeamCity to execute tests and automatically import their test results.
The Visual Studio Test Runner requires Visual Studio Test Agent or Microsoft Visual Studio installed on the build agent.
https://confluence.jetbrains.com/display/TCD10/Visual+Studio+Tests
So you can use VSTests perfectly fine from version 10.
For VSTests, you need to specify a different directory for the config parameter:
teamcity.dotnet.vstest.14.0: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
This file will be available after installing the test agent (or visual studio).
If you still get the error then it means there is something wrong with the tests you have specified, you probably didn't reference to the assemblies correctly, make sure the path and .dll files exist.
There are two problems in the current configuration:
You are using as TestEngine the VSTest that will behave differently than the MsBuild
When you run the Test using the MSBuild this will not generate the .trx files that the Report watcher needs to display the execution results.
So in order to fix your problem you need to change the Test Engine to [MSTest]
This link may help you understand the capabilities of each Test Engine
Choose and configure a test runner
If this helps anyone, in my case this exact error was generated when I wrongly identified the DLL file containing the tests. Instead of "IntegrationTests.dll" I wrote "ItegrationTests.dll". You can identify this issue by looking into the build log where team city says:
Command line params:
and then lists paths to all test DLL files. If it's empty, it means the files you have specified in the build step have not been found.
If you're using VS Test runner, you'll see these lines telling you what happened:
[Step 3/3] No test source files were specified.
[Step 3/3] Process exited with code 1
[Step 3/3] VSTest execution failure
If you're using MS Test runner, you'll see these lines instead:
[Step 3/3] Please specify tests to run, or specify the /publish switch to publish results.
[Step 3/3] For switch syntax, type "MSTest /help"
[Step 3/3] Process exited with code 1
[Step 3/3] MSTest execution failure
Try building the tests using the MSBuild step before the testing step - it seems if it can't find the DLLs containing your tests you get the "No reports found" error

Building Visual Studio 2008 solution from command line

I'm trying to automate the building process for a certain open source project. It will do an update on the SVN directory, use CMake to get a .sln file, and build that. I can successfully do this manually, and do svn and cmake from a batch script, but Now I need to build the solution.
A quick google search revealed:
devenv /build release /project <projname> <solutionfile>.sln
However, that uses the latest version of visual studio (Visual Studio Professional 2011), while the .sln file generated is for Visual C++ Express 2008. I have both versions installed on my computer. Is there a devenv I can use for Visual C++ Express 2008? Or is there a commandline argument to specify which version to use?
UPDATE
I tried using msbuild, but that didn't seem to like building .vcproj files directly, and I didn't want to build ALL the project files by having it build the .sln file. I ended up using this:
"C:\Program Files\Microsoft Visual Studio 9.0\VC\vcpackages\vcbuild.exe" <myproj>.vcproj "Release|Win32"
I think you are using the wrong tool to build this. Rather than trying to drive the IDE from the command line you should simply use msbuild.
msbuild.exe projectname.proj /property:Configuration=Release
In order to set your environment up for the specific version of MSVC you need to call the vcvar.bat file from that specific version. This will set up the necessary environment variables needed by the build tools.
For Visual Studio Express 2008 the IDE is called VCExpress.exe. Also you should probably specify the full path of the program when you have two versions installed:
C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\VCExpress.exe /build release /project <projname> <solutionfile>.sln