Azure DevOps - Builds not showing code coverage when using "dotnet test --collect:'Code Coverage'" - unit-testing

I have a .NET Framework project that is being built on a self hosted Windows build agent.
There is a step to run tests, and that step needs to provide code coverage reports and stats.
When i try using "dotnet test" the step runs and the tests complete, the .coverage files are also generated. When i check the build summary after it's complete i see the standard test results and report, and also the code coverage tab. The code coverage tab has a download link to get the file. There is no code coverage report. There is also a link to "Setup Code Coverage" on the initial build summary screen.
Why is there no code coverage report? and why is the "Setup Code Coverage" link still visible?
This is incredibly frustrating! I must be missing something incredibly obvious, but the docs suggest what i have done is correct.
Using VSTest task rather that dotnet tests results in the same outcome, but runs far slower.
displayName: dotnet test
inputs:
command: test
arguments: '--configuration $(BuildConfiguration) --collect:"Code Coverage"'
workingDirectory: '$(Build.SourcesDirectory)\src'```

I eventually achieved this by using Hugh Lin's answer for help and modifying for my own purposes.
We have Coverlet as a reference in the project, and ReportGenerator installed into Azure DevOps, so that made this a little easier.
I found that we had an issue with a SOAP API reference that was causing the huge performance issues with generating a report. Once I filtered that out with a "classfilter"the process became more manageable. I also found that without the "disable.coverage.autogenerate" variable the "PublishCodeCoverageResults" task will take forever and likely fail as it tries to do the "ReportGenerator" step itself by without the "classfilters". It does this because the "ReportGenerator" is built into the "PublishCodeCoverageResults" step now, but due to having no filters it doesn't work for this scenario.
This is running against a .NET Framework project so there were a few adjustments to the projects needed to ensure "dotnet test" works successfully.
variables:
disable.coverage.autogenerate: 'true'
- task: DotNetCoreCLI#2
displayName: dotnet test
inputs:
command: test
publishTestResults: true
arguments: '/p:CollectCoverage=true /p:CoverletOutputFormat=cobertura --no-restore'
workingDirectory: '$(Build.SourcesDirectory)\src'
configuration: "$(buildConfiguration)"
- task: reportgenerator#4
inputs:
reports: '$(Build.SourcesDirectory)\src\*.UnitTests\coverage.cobertura.xml'
targetdir: '$(Common.TestResultsDirectory)/CoverageReport/'
classfilters: '-NAMESPACE*'
- task: PublishCodeCoverageResults#1
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: '$(Build.SourcesDirectory)\src\*.UnitTests\coverage.cobertura.xml'
reportDirectory: '$(Common.TestResultsDirectory)/CoverageReport/'

By default, the code coverage for the dotnet test task is output to a .codecoverage file, which Azure DevOps does not know how to interpret and only provides as a downloadable file. Code coverage Tab only supports code coverage data in Jacoco or Cobertura formats. So the result of the *.coverage file can not be shown by tables and graphs.
If you want more detailed code coverage report, you need to use coverlet in .Net framework by install the tool during the pipeline and then generate the report. For example in PowerShell script:
dotnet tool install dotnet-reportgenerator --tool-path . --version 4.0.12
dotnet tool install coverlet.console --tool-path . --version 1.4.1
mkdir .\reports
$unitTestFile = gci -Recurse | ?{ $_.FullName -like "*bin\*test*.dll" }
$coverlet = "$pwd\coverlet.exe"
& $coverlet $unitTestFile.FullName --target "dotnet" --targetargs "vstest $($unitTestFile.FullName) --logger:trx" --format "cobertura"
gci -Recurse |
?{ $_.Name -eq "coverage.cobertura.xml"} |
%{ &"$pwd\reportgenerator.exe" "-reports:$($_.FullName)" "-targetdir:reports" "-reportstypes:HTMLInline;HTMLChart" }
Then add Publish code coverage task:
For details, you can refer to the case mentioned in the comment and this ticket.

Related

Module test path not found

I'm trying to configure a GitHub action:
My action contains the job for running the unit test by collecting code coverage. As I see in the log:
Test Run Successful.
Total tests: 336
Passed: 336
Total time: 14.0930 Seconds
Calculating coverage result...
Generating report 'TestResults/coverage.netcoreapp2.1.info'
Nonetheless, after these lines log contains an error message:
/home/runner/.nuget/packages/coverlet.msbuild/2.9.0/build/coverlet.msbuild.targets(31,5): error : Module test path not found [/home/runner/work/ObservableComputations/ObservableComputations/src/ObservableComputations.Test/ObservableComputations.Test.csproj]
The job is failed.
I tried to run
dotnet test --no-build --filter Name~Casting --verbosity normal /p:CollectCoverage=true /p:CoverletOutput=TestResults/ /p:CoverletOutputFormat=lcov
at my local machine (MS Windows) and didn't get this error.
Any help is greatly appreciated.
The reason was in -no-build parameter for dotnet test. It seems coverage collecting requires dotnet test to build itself.

Gitlab test coverage parsing fails

I am trying to get gitlab code coverage parsing working. The server is a local instance of Gitlab 10.4.1-ee. The code coverage tool is lcov via a slightly modified version of this cmake file.
I've entered the regex into the CI Settings as well as in the gitlab ci file to no avail. From what I understand the code coverage will not even attempt to parse if this is not supplied. It did work on one job (out of hundreds) and never again (not sure why). I have supplied both the output and the regex as inputs into http://rubular.com and it seems to parse correctly. I've also fooled around with various iterations of including the single quotes or not or the slashes or not in the regex also to no avail. I can't see where we can get any debugging output or something to show that this step is actually performed.
The project's CI/CD Settings > Test coverage parsing entry:
\bOverall\D+(\d+[.]\d\%)
The job's relevant .gitlab-ci.yml
coverage:
stage: build
image: <redacted>:stable
script:
- mkdir build
- cd build
- cmake -DCMAKE_BUILD_TYPE=Debug .. && make coverage
coverage: '/(?m)\bOverall\D+(\d+[.]\d\%)/'
artifacts:
paths:
- build/coverage/
The job's relevant output:
Overall coverage rate:
lines......: 95.2% (749 of 787 lines)
functions..: 96.5% (110 of 114 functions)
Open ./coverage/index.html in your browser to view the coverage report.
[100%] Built target coverage
Uploading artifacts...
build/coverage/: found 63 matching files
Uploading artifacts to coordinator... ok id=20671 responseStatus=201 Created token=kRnB--qX
Job succeeded
Turns out gitlab's coverage parser is not multi-line. The following regex ended up working lines[\.]+\: (\d+\.\d+)\%. My ci file coverage line ended up being:
coverage: '/lines[\.]+\: (\d+\.\d+)\%/'

What Django TEST_RUNNER supports xunit xml and logging capture?

I'm attempting to set up a new django project, and I've configured TEST_RUNNER in settings.py to be django_nose.NoseTestSuiteRunner.
I chose this test runner because it seems to be the only one I can find that has the following features:
writes xunit xml test report
captures logging/stdout and only displays for failed tests.
However I've heard that nose is unmaintained and I'm having a hard time finding a suitable replacement. The standard test runner doesn't capture logging nor writes xunit as far as I'm able to tell (would love to be proven wrong!)
I run tests like so:
python -m coverage run manage.py test --noinput
python -m coverage report --include="app/*" --show-missing --fail-under=100
python -m coverage xml --include="app/*" -o ./reports/coverage.xml
With this in settings.py:
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
And this setup.cfg:
[nosetests]
verbosity=0
with-xunit=1
xunit-file=./reports/xunit.xml
logging-clear-handlers=1
The last two lines are the real juicy bits I can't seem to find in other test runners. nose captures the logging and clears other logging handlers (eg, the handler that dumps on stdout) so the test runs output is much cleaner (you only see logging for tests that failed).
In other non-django projects I typically use nose2 but django-nose2 project appears to be 6 years old and lacking python3 support??
Please let me know which test runner is the "recommended" one (eg, most popular) with django support, thanks.
I have had success with unittest-xml-reporting:
TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner'
https://github.com/xmlrunner/unittest-xml-reporting#django-support
The output directory can be configured with the TEST_OUTPUT_DIR setting.
You may still use nose runner:
INSTALLED_APPS += ['django_nose']
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
NOSE_ARGS = [
'--with-xunit',
'--xunit-file=nosetests.xml',
'--with-coverage',
'--cover-erase',
'--cover-xml',
'--cover-xml-file=nosecover.xml',
]
So pytest produces some very nice test output. I've unset TEST_RUNNER in settings.py and changed my test script to:
python -m coverage run -m pytest --junitxml=./reports/junit.xml
python -m coverage report --include="app/*" --show-missing --fail-under=100
python -m coverage xml --include="app/*" -o ./reports/coverage.xml
This works, and captures ALL logging output (nose was a little buggy and let one or two logging statements slip through, very strange behavior).
The only thing is that I'm a django novice so I don't know if there are any bad side-effects of not using manage.py test for testing django. Any guidance is appreciated, thanks!

Appveyor runs tests but OpenCover doesn't show coverage

I am building the project and running test cases in appveyor. After successfully executing test cases the coverage data must be uploaded to coveralls.
But in my case no error is thrown but coverage is being recorded.
The details of my project are
.Net Core 1.1.0
Visual Studio 2017
xunit - 2.2.0
OpenCover - 4.6.519
coveralls.net - 0.7.0
My appveyor.yml is as below :
version: 1.0.{build}
os: Visual Studio 2017
skip_tags: true
configuration: Release
environment:
nodejs_version: "0.12"
COVERALLS_REPO_TOKEN:
secure: rstgrtert
cache:
- "%LOCALAPPDATA%\\Yarn"
install:
- npm i -g yarn#0.16.1
- npm i -g typescript typings
- yarn global add typescript typings
- cd ".\Promact.Oauth.Server\src\Promact.Oauth.Server\"
- yarn
- cd..
- cd..
build_script:
- ps: dotnet restore
build:
project: .\Promact.Oauth.Server\Promact.Oauth.Server.sln
verbosity: minimal
test_script:
- cd ".\src\"
- ps: >-
C:\Users\appveyor\.nuget\packages\OpenCover\4.6.519\tools\OpenCover.Console.exe-target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:"test -f netcoreapp1.1 -c Release .\Promact.Oauth.Server.Tests\Promact.Oauth.Server.Tests.csproj" -mergeoutput -hideskipped:File -output:opencover.xml -oldStyle -filter:"+[Promact.Oauth.Server]*Repository -[Promact.Oauth.Server.Tests*]*" -register:user
if(![string]::IsNullOrEmpty($env:COVERALLS_REPO_TOKEN)){
$coveralls = (Resolve-Path "C:\Users\appveyor\.nuget\packages\coveralls.net\0.7.0\tools\csmacnz.coveralls. exe").ToString()
& $coveralls --opencover -i opencover.xml --repoToken
$env:COVERALLS_REPO_TOKEN --commitId $env:APPVEYOR_REPO_COMMIT --commitBranch $env:APPVEYOR_REPO_BRANCH --commitAuthor $env:APPVEYOR_REPO_COMMIT_AUTHOR --commitEmail $env:APPVEYOR_REPO_COMMIT_AUTHOR_EMAIL --commitMessage $env:APPVEYOR_REPO_COMMIT_MESSAGE --jobId $env:APPVEYOR_JOB_ID
}
In appveyor it runs the test and simply shows
Committing...
No results, this could be for a number of reasons. The most common reasons are:
1) missing PDBs for the assemblies that match the filter please review the
output file and refer to the Usage guide (Usage.rtf) about filters.
2) the profiler may not be registered correctly, please refer to the Usage
guide and the -register switch.
Coverage data uploaded to coveralls.
link to the appveyor build

How to run nosetests in jenkins

I have a project which has two folders on same level
/home/ishan/my_repo/jenkins_test/business_logic
/home/ishan/my_repo/jenkins_test/test_cases
test_cases has two files test_fib and test_fact
when I run nosetests --exe at /home/ishan/my_repo/jenkins_test/ it runs correctly showing
....
----------------------------------------------------------------------
Ran 4 tests in 0.036s
OK
I am trying to run these test cases so, I created following script at /home/ishan/my_repo
#!/bin/bash
source /home/ishan/venv/bin/activate
nosetests --exe /home/ishan/sf_shared/my_repo/jenkin_test/
deactivate
When I run it using
source /home/ishan/my_repo/test_runner.sh it shows correct output.
So, I tried to put it in jenksins build step. So, I added the same line
source /home/ishan/my_repo/test_runner.sh in command section of Execute Shell in jenkins.
Now, when I trigger the build using build now it fails saying
Started by user anonymous
Building in workspace /var/lib/jenkins/workspace/jenkins_test
[jenkins_test] $ /bin/sh -xe /tmp/hudson5020664150857393715.sh
+ source /home/ishan/sf_shared/test_runner.sh
/tmp/hudson5020664150857393715.sh: 2: /tmp/hudson5020664150857393715.sh: source: not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I think it doesn't even execute any test cases. It fails long before.
Can you suggest what am I doing wrong?
Maybe this will works:
/home/ishan/venv/bin/nosetests --exe /home/ishan/sf_shared/my_repo/jenkin_test/
Resolved it, issue was with the following line
source /home/ishan/venv/bin/activate
I replaced it source with standard . then it worked. So, my line became
. /home/ishan/venv/bin/activate