We've customized a product which includes their own phpunit test suite. In Jenkins, I have two jobs setup: the first runs our own test suite that covers our customizations, and the second job runs the existing core unit tests.
The core unit tests were not designed to be run on a customized version, so failures are expected. Out of the ~5000 tests, 81 fail. What I'd like to setup in Jenkins, is have the build marked as a failure only if the number of failed tests changes from the previous build.
I've looked at the Performance plugin but the documentation seems sparse and I'm trying to find something that matches our use case.
Any suggestions?
You should have a look at the plugin https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin
It handle a threasolding mechanism (I specified this requirement for the xunit plugin when my team developed it )
expect this helps..
But you want to associates the failure to a change ....
Hum maybe more complex .. have to ask .. if such thing should be developped.
Related
I am looking for a solution where with my every CI build on jenkins i can find with which commit how many and which Unit test cases are broken.
So far i have tried Build Failure Analyzer
But this is not sufficient to get the accurate result.
I am trying the Jacoco-Comparison-Tool. For this there is no Jenkins integration. I am still trying to get a way for this.
Is there any other tools or anything else that can help me to get the UT error/failure reports?
If your project has tests (Unit tests or non-Unit tests), then using JMeter Plugin in Jenkins you can see per build, what all tests passed/failed. https://wiki.jenkins-ci.org/download/attachments/36601991/jmeterV3.jpg?version=1&modificationDate=1260240983000
In Jenkins there's a Test Results Analyzer plugin which also provides some sort of comparison side by side (at class/package level) for N no. of builds with nice charts but it's basically top level info (i.e. it just shows whether this/that test passed/failed in Green/Red color).
There are other plugins (XUnit plugin) that you can try. Also, if you are using SonarQube (analyzing and publishing your tests/results) one can see what happened between two builds (whether the builds failed/passed and to what %).
We have a large Hybris project here and to run all the tests with takes much too long (hours, yes, a large consulting company created that crap). My target is to reduce all the spring based integration tests and replace them by real unit tests.
But when running the tests with the Hybris ant build for one extension (ant alltests -Dtestclasses.extensions=myext) starts a server with the junit tenant also if there are only non Spring based unit tests in that extension. I also tried to use ant unittests but that one does not even executes my tests.
Is there any way to run only the tests annotated with #UnitTest without any server start in an ant run?
PS: I have a hybris 5.1 and 5.3 commerce suite
You should use ant unittests and not ant unit tests:
ant unittests -Dtestclasses.extensions=myext
Note
Running simple unit tests exclusively is not so easy whenever someone uses somewhere Registry.getApplicationContext() in the code under test!
In fact, Registry.getApplicationContext() starts a Hybris instance. If that happens to you, you need to eliminate that particular call to Registry.getApplicationContext() with a better class design and/or mocks.
This is good information. However, in my opinion, even running the unit tests for a single extension is still too much. Unit tests are supposed to be FAST! I should be able to run a single unit test method from within my IDE if I choose to. The whole concept of "red-green testing" is lost if I have to wait for a bunch of non-relevant unit tests to run every time I want to test my refactored code.
Because these tests rely on a runtime environment, there are NO unit tests in Hybris. There are only integration tests because they all rely on a running Hybris system to be executed.
I would like to give some details how to run unittests from within the IDE.
Install IntelliJ
Install Hybris plugun (https://plugins.jetbrains.com/plugin/7525-hybris-integration)
Import the project
Run the UnitTest as any normal developer will do it
Enjoy :)
Is there any tool that analyzes test reports of particular unit test runs and shows differences between them? Basically, I'm interested in a "graph of progress":
12 Aug 2012 10:00: 48/50 tests passed. Failed tests: "MyTest13", "MyTest43".
12 Aug 2012 10:02: 47/50 tests passed. "MyTest13" now passed, but "MyTest2" and "MyTest22" started failing.
NUnit is preferrable, however, unit testing framework is not that important.
I'm looking for a completely automated tool, so that I can set it to run it after each build and instantly look at the results and compare them with previous results. The closest thing I've found is nunit-results and a hand-written batch file to call NUnit (with specified xml report path) and nunit-results as a post-build action. However, html file that it produces is not that informative.
I'm really surprised that noone of the popular unit testing software is capable of storing test run information and analyzing series of runs in bulk. I've tried Resharper, NUnit GUI, Gallio and haven't found anything useful.
I would be glad for a solution that does not require a setup of a complicated CI server. My projects are typically small, but I need a tool like this for every one of them.
I don't know what your threshold is for "complicated CI server", but Jenkins is pretty easy to setup, and with the NUnit Plugin ought to give you what you're after:
This plugin makes it possible to import NUnit reports from each build into Jenkins so they are displayed with a trend graph and details about which tests that failed.
If you are interested in a "Graph of progress", I'd go for a way more simple (IMHO) approach and use NCrunch. It shows you your tests status as you code, without stopping for test runs. See my answer here for more details.
I have a problem with executing tests in JUnit. Imagine you have one test case class with f.e. 100 tests, no test suite and no main program - test case class test the device on com port. JUnit project is in Netbeans. I want to run tests - but not all of them at the same time, i would like to choose tests to run before actual testing.
Once I saw something like that in eclipse - but it wasn't my project and I don't know how it was done and how to do the same thing in netbeans. It was a separate window, poping up before running tests. In this window there were checkboxes with names of methods with #Test annotation and you could choose tests you wanted to run and click run - so it let you to run what you wanted.
Does anyone know how to do it in netbeans? Is it any library or plugin?
Any help will be appreciated.
You can take a look at Run single test from a JUnit class using command-line. It does allow you to specify what test you want to run given a class with multiple test cases in it. Being command-line you can then script your own test suite that runs the specific ones you want.
I also noticed your other question Junit: changing sequence of test running. With the scripting approach you can actually control the order of your testing.
This approach does not take advantage of Eclipse's or NetBean's JUnit test runners though, so it is a very specific workaround.
Netbeans nowadays support running single tests:
I want to follow TDD, but the command grails test-app CUT needs almost a minute to run due to Resolving dependencies... and Resolving new plugins. Please wait... ...
Each of those two stages takes about 20 seconds to complete while the tests only take up some seconds.
(I am unsure if this has any effect on the performance, but I am using dependency resolution via BuildConfig.groovy - and want to stick with it.)
How can I have grails only execute the tests any maybe skip the process of resolving?
How else could I speed up the process? (Note that grails interactive is unable to influence the speed of resolving.)
I had a similar issue and solved it by not using *-SNAPHOT versions of any plugins. I downgraded to the latest non-SNAPSHOT release and cut "resolving dependencies" from 10 seconds to 1 second.
Ideas:
Try removing (or to be safe moving) the directory /.ivy2/cache. The next time you do a 'run-app' all the dependencies will be downloaded again from scratch. After doing this I got my 'Resolving Dependencies...' time down by about 5 seconds.
There are some more tips on how to fully clean your directories here A full clean may help if you have some inconsistent files etc.
Try turning the logging on in BuildConfig.groovy by setting log to "info" in the grails.project.dependency.resolution section. This can give you a better idea of which dependencies are taking the longest.
Make sure your .ivy2 directory is on your local machine. See here for more info
In Grails 2 there's a new variant of the old (now deprecated) 'interactive' command. In order to start it, one must start grails without any arguments (i.e. grails <ENTER>).
Running test-app from there seems to skip dependecy resolution which ultimately makes tests run much faster now (~40 seconds less in the case mentioned).
You should write your unit tests in a way that you can run them directly from the IDE. I like looking at the green bar. For example in STS/Eclipse, just do "Run As->Junit Test". If the test requires Grails to be running, it's not an unit test anymore (it's an integration test).
I am going to have to back up FlareCoder on this. Too many Grails developers get lazy using Grails specific unit tests or worse, make everything an integration test. This is fine if your project is relatively small and your team does not mind Grails to start up every time but it does kind of fly in the face of true TDD.
Once you understand the full power of Groovy outside of Grails, you should try to write unit tests without depending on Grails. The true spirit of a unit test is not requiring a framework. Groovy on its own has many ways to stub/mock classes that don't require a long startup time. Then your unit tests can run individually and as a whole very fast. I do TDD this way in IntelliJ IDEA on a method level that is very fast.
It is NOT true that mocking in Grails requires Grails mocking ALL the time. Sometimes it is harder than other times to achieve this but remember, Grails is simply an abstraction of many cool technologies using some Groovy metaprogramming that allow quick development. If they aren't running like you expect, dig in and understand them so you can remove anything Grails is doing that you don't need.