We use TeamCity 7 (upgrade to 8 is possible) for continuous integration and we set ourselves a target of unit test coverage 90%. I know how to fail the build if the coverage is lower, but I'd not like to do so, as a missing test will slow down all the development.
On the other hand, I'd like to have clear visibility on the build overview page that the coverage is low - the only option I see is a service message like this one:
##teamcity[buildStatus status='SUCCESS' text='WARN: Test coverage only 89% {build.status.text}']
But that won't send any notification. Do you have any other suggestions, please?
Set the coverage html as artifact and link to it from Teamcity. IE setup a new tab, it will look something like this.
Related
Currently, I am using jacoco report tool to know how much coverage for my system. But this system is quite old and has many existing Fitnesse tests. I am newbie on this system and want to learn it by reading its Fitnesse test. My problem is I dont know what Fitnesse test is for what specific class.Beside that When I see a class is covered and I dont know which Fitnesse test covered this class. My system has more than 500 Fitnesse Tests.
JaCoCo does not provide this out of the box.
There are some third-party JaCoCo integrations such as SonarQube that allow to obtain information about relation between JUnit or TestNG test to code. Don't know much about Fitnesse, so can't tell whether this will work with it or not.
However referring to the same link: general principle to obtain such information using JaCoCo (and that's how SonarQube does) - is to measure coverage separately for each test and save data into place dedicated to this test.
Also IDEs are able to show coverage (there are EclEmma plugin for Eclipse based on JaCoCo and other tools for other IDEs), so if you're able to run Fitneese test in IDE, then this principle can be applied even manually by running one test after another within IDE.
Also have a look at another code coverage tool for Java that is commercial and named Clover.
I am looking for a solution where with my every CI build on jenkins i can find with which commit how many and which Unit test cases are broken.
So far i have tried Build Failure Analyzer
But this is not sufficient to get the accurate result.
I am trying the Jacoco-Comparison-Tool. For this there is no Jenkins integration. I am still trying to get a way for this.
Is there any other tools or anything else that can help me to get the UT error/failure reports?
If your project has tests (Unit tests or non-Unit tests), then using JMeter Plugin in Jenkins you can see per build, what all tests passed/failed. https://wiki.jenkins-ci.org/download/attachments/36601991/jmeterV3.jpg?version=1&modificationDate=1260240983000
In Jenkins there's a Test Results Analyzer plugin which also provides some sort of comparison side by side (at class/package level) for N no. of builds with nice charts but it's basically top level info (i.e. it just shows whether this/that test passed/failed in Green/Red color).
There are other plugins (XUnit plugin) that you can try. Also, if you are using SonarQube (analyzing and publishing your tests/results) one can see what happened between two builds (whether the builds failed/passed and to what %).
We've customized a product which includes their own phpunit test suite. In Jenkins, I have two jobs setup: the first runs our own test suite that covers our customizations, and the second job runs the existing core unit tests.
The core unit tests were not designed to be run on a customized version, so failures are expected. Out of the ~5000 tests, 81 fail. What I'd like to setup in Jenkins, is have the build marked as a failure only if the number of failed tests changes from the previous build.
I've looked at the Performance plugin but the documentation seems sparse and I'm trying to find something that matches our use case.
Any suggestions?
You should have a look at the plugin https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin
It handle a threasolding mechanism (I specified this requirement for the xunit plugin when my team developed it )
expect this helps..
But you want to associates the failure to a change ....
Hum maybe more complex .. have to ask .. if such thing should be developped.
I am trying to measure branch coverage of unit tests for a large Grails application. I am using JaCoCo, Emma and IDEA to collect the metrics from inside IntelliJ, I am getting the following:
JaCoCo (no metrics are shown even for line coverage)
Emma (produces method and line coverage)
IDEA (produces class, method and line coverage)
I am mostly interested in JaCoCo as it should give me Branch Coverage by default. Could someone point me to some tips on how to troubleshoot this?
Actually IntelliJ code coverage tool supports branch coverage though it does not show the results on the summary. Check this article to see how it can be configured and how you can check your branch coverage: https://confluence.jetbrains.com/display/IDEADEV/IDEA+Coverage+Runner
The key is to use Tracing instead of Sampling.
Is there any tool that analyzes test reports of particular unit test runs and shows differences between them? Basically, I'm interested in a "graph of progress":
12 Aug 2012 10:00: 48/50 tests passed. Failed tests: "MyTest13", "MyTest43".
12 Aug 2012 10:02: 47/50 tests passed. "MyTest13" now passed, but "MyTest2" and "MyTest22" started failing.
NUnit is preferrable, however, unit testing framework is not that important.
I'm looking for a completely automated tool, so that I can set it to run it after each build and instantly look at the results and compare them with previous results. The closest thing I've found is nunit-results and a hand-written batch file to call NUnit (with specified xml report path) and nunit-results as a post-build action. However, html file that it produces is not that informative.
I'm really surprised that noone of the popular unit testing software is capable of storing test run information and analyzing series of runs in bulk. I've tried Resharper, NUnit GUI, Gallio and haven't found anything useful.
I would be glad for a solution that does not require a setup of a complicated CI server. My projects are typically small, but I need a tool like this for every one of them.
I don't know what your threshold is for "complicated CI server", but Jenkins is pretty easy to setup, and with the NUnit Plugin ought to give you what you're after:
This plugin makes it possible to import NUnit reports from each build into Jenkins so they are displayed with a trend graph and details about which tests that failed.
If you are interested in a "Graph of progress", I'd go for a way more simple (IMHO) approach and use NCrunch. It shows you your tests status as you code, without stopping for test runs. See my answer here for more details.