We use VSTS with the newest SonarQube tasks and Sonarqube 5.6.1
In Sonarqube we see all the unit test coverage results, except for one item: The nr of unit tests. How/what do we need to configure to have the nr of unit tests also shown in Sonar Qube?
Per SonarC# documentation, you need to import the Unit Test Execution Results, using the applicable property (for example sonar.cs.vstest.reportsPath). The trick is to set the appropriate value, which is not always straightforward in automated environments (e.g. VSTS).
Pending planned improvements with SONARMSBRU-231, you may want to try the workaround mentioned in that ticket:
/d:sonar.cs.vstest.reportsPaths=..\**\TestResults\**\*.trx
(under Advanced, Additional Settings , in the Prepare the SonarQube analysis build step)
Related
For Java I know the possibility to merge test coverage results on build level by specifying the same path of JaCoCo reports (see SonarQube: Multiple unit test and code coverage result files). This might be transported to SonarQube.
But is it possible to make this on SonarQube level?
I mean from different build servers or different jobs build and test software and combine coverage results at SonarQube side (perhaps by marking the SW version or any kind of given label)?
For me it would be usefull to combine integration and unit tests.
You can combine the result of multiple jobs. You can create two coverage folders, e.g.
- coverage-unit
- coverage-integration
and use the resulting lcov files, e.g.
sonar.javascript.lcov.reportPaths=coverage-unit/lcov.info,coverage-integration/lcov.info
Currently it is not possible to "amend" coverage to an existing analysis. You have to orchestrate your build pipeline so that all kind of coverage reports are produced before you actually starts the SonarQube analysis.
I am looking for a solution where with my every CI build on jenkins i can find with which commit how many and which Unit test cases are broken.
So far i have tried Build Failure Analyzer
But this is not sufficient to get the accurate result.
I am trying the Jacoco-Comparison-Tool. For this there is no Jenkins integration. I am still trying to get a way for this.
Is there any other tools or anything else that can help me to get the UT error/failure reports?
If your project has tests (Unit tests or non-Unit tests), then using JMeter Plugin in Jenkins you can see per build, what all tests passed/failed. https://wiki.jenkins-ci.org/download/attachments/36601991/jmeterV3.jpg?version=1&modificationDate=1260240983000
In Jenkins there's a Test Results Analyzer plugin which also provides some sort of comparison side by side (at class/package level) for N no. of builds with nice charts but it's basically top level info (i.e. it just shows whether this/that test passed/failed in Green/Red color).
There are other plugins (XUnit plugin) that you can try. Also, if you are using SonarQube (analyzing and publishing your tests/results) one can see what happened between two builds (whether the builds failed/passed and to what %).
We've customized a product which includes their own phpunit test suite. In Jenkins, I have two jobs setup: the first runs our own test suite that covers our customizations, and the second job runs the existing core unit tests.
The core unit tests were not designed to be run on a customized version, so failures are expected. Out of the ~5000 tests, 81 fail. What I'd like to setup in Jenkins, is have the build marked as a failure only if the number of failed tests changes from the previous build.
I've looked at the Performance plugin but the documentation seems sparse and I'm trying to find something that matches our use case.
Any suggestions?
You should have a look at the plugin https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin
It handle a threasolding mechanism (I specified this requirement for the xunit plugin when my team developed it )
expect this helps..
But you want to associates the failure to a change ....
Hum maybe more complex .. have to ask .. if such thing should be developped.
When I scan a multi-modular project for Sonar with the same projectBaseDir the unit test count is wrong. My groovy app has 80 unit tests, my java 0 and my javascript 0. Every time I scan my project, the unit test count ends up 240. If I take the java or the js modules out, it goes down to 160 and down to 80 if I remove the other two modules entirely. How can I have a modular project that shares the same root and won't double/triple count the unit tests? I shouldn't have to have a separate Sonar project or complete module sub-directory for each language I scan in the project.
Sonar 3.7.4, framework 1.4, Sonar-Runner 2.3, sonar-groovy-plugin-1.0-spantree-SNAPSHOT.jar
sonar.projectKey=com.me:myapp
sonar.projectName=MyApp
sonar.projectVersion=1.0
sonar.projectDescription=
sonar.modules=module-java, module-js,module-grvy
module-grvy.sonar.projectName=MyApp - Grails
module-grvy.sonar.language=grvy
module-grvy.sonar.sources=src/groovy
module-grvy.sonar.tests=test/unit
module-grvy.sonar.projectBaseDir=.
module-java.sonar.projectName=MyApp - Java
module-java.sonar.language=java
module-java.sonar.sources=src/java
module-java.sonar.tests=
module-java.sonar.projectBaseDir=.
module-js.sonar.projectName=MyApp - JavaScript
module-js.sonar.language=js
module-js.sonar.sources=web-app
module-js.sonar.tests=
module-js.sonar.exclusions=**/built/*.js
module-js.sonar.projectBaseDir=.
sonar.sourceEncoding=UTF-8
sonar.dynamicAnalysis=reuseReports
sonar.surefire.reportsPath=target/test-reports
sonar.core.codeCoveragePlugin=cobertura
sonar.cobertura.reportPath=target/test-reports/cobertura/coverage.xml
You need to use modules for reporting unit test results to sonar if you need to see the unit test count appropriate to a module.
For explaining this, I will take a .NET example and you can relate it to any language for achieving similar working results.
Solution: Lets say I have three modules(dlls) as below.
MVP.Unit.Test.1.dll - 5 tests
MVP.Unit.Test.2.dll - 8 tests
MVP.Unit.Test.3.dll - 2 tests
I need to run the unit tests in these dlls separately so that separate unit test reports are generated for these test dlls. And the unit test reports are as follows.
MVP.Unit.Test.1.trx (.trx is the extension of unit test report in .NET world)
MVP.Unit.Test.2.trx
MVP.Unit.Test.3.trx
Now that I have separate unit test reports for different modules, we need to report this to sonar using properties file as following.
MVP.Unit.Test.1.sonar.cs.vstest.reportPaths=C:/Test/MVP.Unit.Test.1.trx
MVP.Unit.Test.2.sonar.cs.vstest.reportPaths=C:/Test/MVP.Unit.Test.2.trx
MVP.Unit.Test.3.sonar.cs.vstest.reportPaths=C:/Test/MVP.Unit.Test.3.trx
What the above configuration will do is, it will tell Sonar to properly map the unit test results with correct modules so that we get nice segmentation of unit tests according to their modules.
There you go. Now you'll be able to see the unit test count in Sonar as per the modules.
Is there any tool that analyzes test reports of particular unit test runs and shows differences between them? Basically, I'm interested in a "graph of progress":
12 Aug 2012 10:00: 48/50 tests passed. Failed tests: "MyTest13", "MyTest43".
12 Aug 2012 10:02: 47/50 tests passed. "MyTest13" now passed, but "MyTest2" and "MyTest22" started failing.
NUnit is preferrable, however, unit testing framework is not that important.
I'm looking for a completely automated tool, so that I can set it to run it after each build and instantly look at the results and compare them with previous results. The closest thing I've found is nunit-results and a hand-written batch file to call NUnit (with specified xml report path) and nunit-results as a post-build action. However, html file that it produces is not that informative.
I'm really surprised that noone of the popular unit testing software is capable of storing test run information and analyzing series of runs in bulk. I've tried Resharper, NUnit GUI, Gallio and haven't found anything useful.
I would be glad for a solution that does not require a setup of a complicated CI server. My projects are typically small, but I need a tool like this for every one of them.
I don't know what your threshold is for "complicated CI server", but Jenkins is pretty easy to setup, and with the NUnit Plugin ought to give you what you're after:
This plugin makes it possible to import NUnit reports from each build into Jenkins so they are displayed with a trend graph and details about which tests that failed.
If you are interested in a "Graph of progress", I'd go for a way more simple (IMHO) approach and use NCrunch. It shows you your tests status as you code, without stopping for test runs. See my answer here for more details.