Is there any tool that analyzes test reports of particular unit test runs and shows differences between them? Basically, I'm interested in a "graph of progress":
12 Aug 2012 10:00: 48/50 tests passed. Failed tests: "MyTest13", "MyTest43".
12 Aug 2012 10:02: 47/50 tests passed. "MyTest13" now passed, but "MyTest2" and "MyTest22" started failing.
NUnit is preferrable, however, unit testing framework is not that important.
I'm looking for a completely automated tool, so that I can set it to run it after each build and instantly look at the results and compare them with previous results. The closest thing I've found is nunit-results and a hand-written batch file to call NUnit (with specified xml report path) and nunit-results as a post-build action. However, html file that it produces is not that informative.
I'm really surprised that noone of the popular unit testing software is capable of storing test run information and analyzing series of runs in bulk. I've tried Resharper, NUnit GUI, Gallio and haven't found anything useful.
I would be glad for a solution that does not require a setup of a complicated CI server. My projects are typically small, but I need a tool like this for every one of them.
I don't know what your threshold is for "complicated CI server", but Jenkins is pretty easy to setup, and with the NUnit Plugin ought to give you what you're after:
This plugin makes it possible to import NUnit reports from each build into Jenkins so they are displayed with a trend graph and details about which tests that failed.
If you are interested in a "Graph of progress", I'd go for a way more simple (IMHO) approach and use NCrunch. It shows you your tests status as you code, without stopping for test runs. See my answer here for more details.
Related
Currently, I am using jacoco report tool to know how much coverage for my system. But this system is quite old and has many existing Fitnesse tests. I am newbie on this system and want to learn it by reading its Fitnesse test. My problem is I dont know what Fitnesse test is for what specific class.Beside that When I see a class is covered and I dont know which Fitnesse test covered this class. My system has more than 500 Fitnesse Tests.
JaCoCo does not provide this out of the box.
There are some third-party JaCoCo integrations such as SonarQube that allow to obtain information about relation between JUnit or TestNG test to code. Don't know much about Fitnesse, so can't tell whether this will work with it or not.
However referring to the same link: general principle to obtain such information using JaCoCo (and that's how SonarQube does) - is to measure coverage separately for each test and save data into place dedicated to this test.
Also IDEs are able to show coverage (there are EclEmma plugin for Eclipse based on JaCoCo and other tools for other IDEs), so if you're able to run Fitneese test in IDE, then this principle can be applied even manually by running one test after another within IDE.
Also have a look at another code coverage tool for Java that is commercial and named Clover.
I have test scenario where its possible that intermittently all tests within a test run will skip (intentionally), in this case I want Jenkins to mark the build as UNSTABLE. At the moment it marks the job as PASSED, which causes issues when we want quick visual feedback (via dashboard) as to what jobs need attention as all we see are green jobs.
Background:
Tests written in python 2.7.
Test runner used is Nose.
Test results are output using ‘—with-xunit’ flag in nose.
Its a single job that's sole purpose is to run the tests.
Hoping there is a solution as I’m yet to find an obvious one. cheers.
I would suggest a post-build Groovy script to investigate the results and mark it as unstable. Take a look at this similar question:
Jenkins Groovy Script finding null testResultAction for a successful run
The plug-in page (which contains a lot of examples):
https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Postbuild+Plugin
I am looking for a solution where with my every CI build on jenkins i can find with which commit how many and which Unit test cases are broken.
So far i have tried Build Failure Analyzer
But this is not sufficient to get the accurate result.
I am trying the Jacoco-Comparison-Tool. For this there is no Jenkins integration. I am still trying to get a way for this.
Is there any other tools or anything else that can help me to get the UT error/failure reports?
If your project has tests (Unit tests or non-Unit tests), then using JMeter Plugin in Jenkins you can see per build, what all tests passed/failed. https://wiki.jenkins-ci.org/download/attachments/36601991/jmeterV3.jpg?version=1&modificationDate=1260240983000
In Jenkins there's a Test Results Analyzer plugin which also provides some sort of comparison side by side (at class/package level) for N no. of builds with nice charts but it's basically top level info (i.e. it just shows whether this/that test passed/failed in Green/Red color).
There are other plugins (XUnit plugin) that you can try. Also, if you are using SonarQube (analyzing and publishing your tests/results) one can see what happened between two builds (whether the builds failed/passed and to what %).
We've customized a product which includes their own phpunit test suite. In Jenkins, I have two jobs setup: the first runs our own test suite that covers our customizations, and the second job runs the existing core unit tests.
The core unit tests were not designed to be run on a customized version, so failures are expected. Out of the ~5000 tests, 81 fail. What I'd like to setup in Jenkins, is have the build marked as a failure only if the number of failed tests changes from the previous build.
I've looked at the Performance plugin but the documentation seems sparse and I'm trying to find something that matches our use case.
Any suggestions?
You should have a look at the plugin https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin
It handle a threasolding mechanism (I specified this requirement for the xunit plugin when my team developed it )
expect this helps..
But you want to associates the failure to a change ....
Hum maybe more complex .. have to ask .. if such thing should be developped.
This question already has answers here:
Using Post-Build Event To Execute Unit Tests With MS Test in .NET 2.0+
(3 answers)
Closed 8 years ago.
i use VS2013. I have a "normal project" and a testproject with some testcases to the "normal project". I can run the tests in test project - so far so good. But now, I want to run the tests automatically, when I rebuild my "normal project". Any ideas?
Thanks.
As was pointed out in the comments to my other answer, it is possible to add a postbuild step which will invoke the tests from the command line. The details can be found in this question.
Whilst this might be a suitable solution, I believe it is inferior to the continuous test runner approach for several reasons.
feedback is not integrated nice into the IDE in a test runner window, allowing for simple visual feed back and/or running of individual or groups of tests
Tests only run at build time. Continuous runners will run when changes are detected
All tests will be run on every save. Continuous runners can identify only the tests that have been affected by the change and run only those tests.
running individual tests will require using a different runner
running specified subsets of tests (ie excluding integration tests) will require changing the postbuild step. NCrunch can be configured to run filtered sets of tests.
Having said all that some of the runners are commercial products that are not cheap, (although some are Open Source and free). In the end you pays your money, you takes your choice.
I don't know of anyway to do this in natively in VS but you should look into a tool like NCrunch, a continuous test runner, which will run your tests not only when you build your 'normal' project but also when you make any change to the project.
There are other continuous test runners out there, like MightyMoose, the source of which is here I believe
I'm in no way affiliated with NCrunch (or mighty moose), but I think its awesome and can't imagine life without it now.
Its not free though so whether its useful for you I can't say.