As part of the TFS Build we execute tests that generate trx files. I would like to execute on these set of files code coverage analysis using OpenCover, without re-executing the Unit Tests.
Is there a way to do so?
Thanks
This is a little command-line & duck-tape but it gets the OpenCover and Test results in 1 pass, and publishes test results and coverage to TFS or Visual Studio Team Services (previously VSO).
Create .bat file Run-Tests.bat that calls vstest.console.exe to test your code with the /logger:Trx option
This creates a .trx file that can be published to TFS
Run the tests
OpenCover.Console.exe -target:"Run-Tests.bat" -output:"OpenCover.xml"
Run OpenCoverToCoberturaConverter:
OpenCoverToCoberturaConverter.exe -input:OpenCover.xml -output:Cobertura.xml
Publish the test results and coverage to TFS:
Unfortunately not - OpenCover needs to instrument your code as it runs in order to determine which bits of code were executed.
Related
I cannot get coverage reporting to work within SonarQube. I have a C++ project for which I am using the build-wrapper-linux-x86-64 along with the sonar-scanner. The basic static analysis for the source code seems to work but there is nothing about test code coverage reported within SonarQube.
As part of the same workflow I am using lcov and genhtml to make a unit test coverage report, so I am confident that most of the code coverage steps are being correctly executed. When I manually view the .gcov files I can see run counts in the first column, so there is data there.
I have my code organised into modules. The sonar-project.properties file includes the following:
# List of the module identifiers
sonar.modules=Module1,Module2
# Path is relative to the sonar-project.properties file. Replace "\" by "/" on Windows.
# This property is optional if sonar.modules is set.
sonar.sources=./Sources,./Tests
HeliosEmulator.sonar.sources=./Application,./Sources,./Tests
sonar.cfamily.build-wrapper-output=build_output
# Existing reports
sonar.cfamily.build-wrapper-output=build_output
#sonar.cfamily.cppunit.reportsPath=junit
sonar.cfamily.gcov.reportsPath=.
#sonar.cxx.cppcheck.reportPath=cppcheck-result-1.xml
#sonar.cxx.xunit.reportPath=cpputest_*.xml
sonar.junit.reportPaths=junit
I would also like to get the unit test results displayed under the Sonar tools. As I am using the CppUTest framework I do not have an xunit or junit test output at present though. This can be dealt with as a separate issue but as I am unable to found much documentation of how to use the cfamily scanner online I do not know if the tests not being listed is relevant.
I had forgotten to setup my CI system correctly. The .gcov files did not exist for the job that was running the sonar-scanner. They only existed in the testing job that generated the coverage report. No files in the scanner job mean it cannot make a coverage report.
When I set the GitLab CI system I am using to keep the .gcov files as artefacts the coverage reporting suddenly started working.
The .gcov files were generated by a test job and need to be transferred to the sonar-scanner job via the artefact store. This is because GitLab CI does not share a work area between dependent jobs and you have to explicitly say what files must be copied.
I have this build :
It works fine. The only issue is that the Test Results are overridden. So I actually end up with the test results for the last test project executed.
This is executed by build engine;
C:\Program Files\dotnet\dotnet.exe test C:/agent/_work/4/s/test/Services.UnitTests/project.json --configuration release -xml ./TEST-tle.xml
C:\Program Files\dotnet\dotnet.exe test C:/agent/_work/4/s/test/Web.UnitTests/project.json --configuration release -xml ./TEST-tle.xml
What could help:
1) having "dotnet test" generate XML output file - did not find a way how to do that
2) Use a variable for -xml output file in Build Task. That variable could be a random string/number or just a project name being tested - like what Build engine feeds to "dotnet.exe test". No way how to do that.
Any ideas? Thanks.
I think that, although you're running the task against all of the projects in one go, as the .Net Core (Preview) task doesn't have a working directory, that the test results are being generated at solution root (or similar) and done for each project in turn.
I set mine up using simple command line tasks...
Tool: dotnet
Arguments: test -xml testresults.xml
Working folder: {insert the folder for the project to test here}
These work fine but I have one set up for each project. You could try creating a task for each library and adding the full path to the test results argument (or name them appropriately as starain suggested).
This feels like a minor bug to me.
Based on my test, it doesn’t recognize the date variable as Build Number.
To deal with this issue, you can add another .Net Core (Test) step to run xunit test with different result file.
For example:
This is the command I use to run all NUnit tests from specified library
vstest.console.exe "PATH_TO_REPOSITORY\Tests\terminalBaseTests\bin\debug\terminalBaseTests.dll" /logger:trx /TestAdapterPath:"PATH_TO_REPOSITORY"
This dll contains 27 tests but I see that they launched for some reason 3 times and the resulting message says that 81 tests were passed
Demarch provided the correct answer in the comments, for visibility and whenever other people are stumbling upon this issue I will state it here (it took me way too long to figure this out):
Symptom:
VSTest.Console.exe is running tests twice or even multiple times when NUnit, Xunit or another testrunner is used than the native one.
Cause:
The path to the TestAdapter is not correctly (to $(Build.SourcesDirectory) for example) set or not present. This is letting the runner scan all folders for test adapters. When there are multiple present, all testadapters will start a testrun causing all tests run multiple times.
Solution:
Redirect the path to the testadapter to the folder your packages are in.
If you run from the commandline set the following parameter for VSTest.Console.exe to something like this: /TestAdapterPath:"{solutionfolder}\packages"
If you run it in TFS, open the "Advanced Executions Options" section of the Visual Studio Test build step and set the following parameter to something like this: $(Build.SourcesDirectory)\packages
I have created a TFS2013 Build Definition using the template TfvcTemplate.12.xaml
I have specified a test run using VSTestRunner and enabled code coverage.
I am integrating this build with sonar analysis by specifying pre-build and post-test execution script.
Prebuild script arguments: begin /name:PrjName /key:PrjKey /version:1.0 /d:sonar.cs.vstest.reportsPaths="tst*.trx"
I have the "Unit Test Coverage" widget on my sonar dashboard.
It shows Unit Test Coverage %
However, it does not show the unit tests (ie how many tests were run, how many failed ,etc).
I looked in the build output. There is a "tst" folder, however it is empty.
I cannot find the trx files.
I believe that either the trx files are not properly generated or
I am not setting the "sonar.cs.vstest.reportsPaths" correctly.
Please help !!
Relative paths are not well supported: Specify an absolute path wildcard to your *.trx reports. See https://jira.sonarsource.com/browse/SONARMSBRU-100 for details on the bug.
Note that you probably can use the TFS 2013 environment variables to construct this absolute path wildcard: https://msdn.microsoft.com/en-us/library/hh850448.aspx#env_vars
We have created a build definition in TFS 2013 that runs unit tests and then calls a PowerShell script once completed.
What we would like is the ability to pass the results of the unit test (passed/failed) to the PowerShell script so that a check can be done to verify if the unit tests passed.
Is there a parameter that can be used in the 'Post-test script path' setting or a possible workaround?
You should be able to use the predefined variable/property combo BuildDetail.TestStatus.