I have test scenario where its possible that intermittently all tests within a test run will skip (intentionally), in this case I want Jenkins to mark the build as UNSTABLE. At the moment it marks the job as PASSED, which causes issues when we want quick visual feedback (via dashboard) as to what jobs need attention as all we see are green jobs.
Background:
Tests written in python 2.7.
Test runner used is Nose.
Test results are output using ‘—with-xunit’ flag in nose.
Its a single job that's sole purpose is to run the tests.
Hoping there is a solution as I’m yet to find an obvious one. cheers.
I would suggest a post-build Groovy script to investigate the results and mark it as unstable. Take a look at this similar question:
Jenkins Groovy Script finding null testResultAction for a successful run
The plug-in page (which contains a lot of examples):
https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Postbuild+Plugin
Related
As we only run our unit tests once a day it can happen that multiple changes led to a failing test. We then go into the changes list and trigger the tests for each change until we find the one responsible for breaking the test.
How can we automate this? We want TeamCity to run the unit tests again for the different changes (some binary search logic would be a bonus) until it finds the culprit.
How would you call this feature? I'm looking at the options to enable this but haven't had any luck so far.
Thanks for input and pointers.
I've developed a TC plugin to deal with this. See https://github.com/sferencik/SinCity. Read the docs and see if it suits you. I'm happy to help you further if you need.
The docs also mention the only other alternative I'm aware of: https://github.com/tkirill/tc-bisect. That has the bisect functionality ("binary search logic") but I'm not sure what state it's in.
When using WebStorms as a test runner every unit test is run. Is there a way to specify running only one test? Even only running one test file would be better than the current solution of running all of them at once. Is there a way to do this?
I'm using Mocha.
not currently possible, please vote for WEB-10067
You can double up the i on it of d on describe and the runner will run only that test/suite. If you prefix it with x it will exclude it.
There is a plugin called ddescribe that gives you a gui for this.
You can use the --grep <pattern> command-line option in the Extra Mocha options box on the Mocha "Run/Debug Configurations" screen. For example, my Extra Mocha options line says:
--timeout 5000 --grep findRow
All of your test *.js files, and the files they require, still get loaded, but the only tests that get run are the ones that match that pattern. So if the parts you don't want to execute are tests, this helps you a lot. If the slow parts of your process automatically get executed when your other modules get loaded with require, this won't solve that problem. You also need to go into the configuration options to change the every time you want to run tests matching a different pattern, but this is quick enough that it definitely saves me time vs. letting all my passing tests run every time I want to debug one failing test.
You can run the tests within a scope when you have a Mocha config setting by using .only either on the describe or on the it clauses
I had some problems getting it to work all the time, but when it went crazy and kept running all my tests and ignoring the .only or .skip I added to the extra mocha options the path to one of the files containing unit tests just like in the example for node setup and suddenly the .only feature started to work again regardless of the file the tests were situated in.
I have a huge group of unit tests that I'm running against our software where I work. I'm running them using IntelliJ IDEA. I'm using Spock and Groovy to make these tests, but they are, in turn, using JUnit. So anything that applies to JUnit should apply here as well... (in theory)
In order to figure out what coverage is I'm right clicking on the root of the code I want to run and simply selecting "Test in 'example' with Coverage".
IntelliJ then proceeds to try and run the 270 or some odd tests we have in this part of the project. The problem is that it isn't running a couple of them and I can't figuring out why for the life of me.
I've googled the issue and turned up nothing substantial. The list of tests in IntelliJ only tells me that these couple of tests didn't start, it doesn't give a reason at all. I tried checking the logs, but all it says about these particular tests is that it's trying to open them, no comment on how successful they were or were not, and why they aren't working.
If someone could just point me in a useful direction? I need to expand my test bases coverage and being able to see what code is actually covered is a pretty critical...
Some Clarification:
The tests can be run on their own, and they do computer coverage for themselves if you do so. They only can't be run when I run them in a batch.
I'm pretty sure it's not a display issue because I get several different reports telling me that the tests were not run. So, more than just the frozen yellow spinner it also warns me with a red "attention" bubble that the tests did not run.
I've become responsible for cleaning up an old unit-testing environment. The existing environment contains a ton of executables (1000+ shell scripts, compiled binaries, etc) and each one returns code 0 or 1 and some output depending on the results. Tests can also time out. A set of PERL scripts goes through, runs each executable file, and collects the results into a big XML file which gets rendered into a web page. This system works great, but isn't very extensible or fast.
In addition to cleaning/speeding this up, I would like to implement concurrent testing. Right now the tests run one at a time. Many of the tests require resource locks (ports, files, etc) and there's no listing of which are safe to run simultaneously. An option here is to run each one in a VM of some kind.
Is there a framework or tool designed for this type of situation? What's the best way to approach it if I have to write my own brand new system? My limitation is that I cannot change the 1000+ test executables. I was thinking something like PyUnit with Python unit tests that use subprocess or similar to execute the existing tests and convert the results into a Pythonic format. Is this overkill? Could such a system support isolation of processes to avoid deadlocks and race conditions between tests? In a worst-case, I was thinking of using Python to SSH into one of several VM's to run the tests.
Any help or guidance would be appreciated.
Edit: Python isn't strictly necessary. Another solution might just break the test set into several M-sized chunks and run each test in an independent VM over SSH until all M-sized chunks are done. I'm hoping there's a tool out there that solves this problem.
There is no out-of-the-box or customize it for your needs solution of which I am aware to solve the problem you are facing.
Looking at your problem, there are several distinct needs which standout:
Test Tagging
Test Execution
Test Result Capture
The first issue you need to address is how are you going to identify and track the tests that you can execute in a given environment, concurrently, etc.
If you were using nose (i.e. is nicer testing for python), you would be able to use the Attribute selector plugin to tag the tests with various attributes.
nose would also assist you in your test execution and when coupled with test tagging, would allow you to run tests in parallel, etc. Using nose, you should be able to run an external executable and assert based on its status code.
The final problem you face is how to capture test output in a proprietary format and translate it to a format that can be ingested to readily available tools. Again, I believe nose could help you out here. You could build a nose plugin that would take your proprietary format and translate it to XUnit format and report results that way.
With all of the above in mind, here is how I would tackle this situation:
Create a test wrapper class based on nose which
Can be tagged
Execute a program and capture result output
Translate that output to XUnit
Create a wrapper for each test
Figure out how to automate this process, because it is going to be tedious
Build a test execution harness, which
Spins Up one or more VMs
Loads and runs a test wrapper
Capture the results
Is there any tool that analyzes test reports of particular unit test runs and shows differences between them? Basically, I'm interested in a "graph of progress":
12 Aug 2012 10:00: 48/50 tests passed. Failed tests: "MyTest13", "MyTest43".
12 Aug 2012 10:02: 47/50 tests passed. "MyTest13" now passed, but "MyTest2" and "MyTest22" started failing.
NUnit is preferrable, however, unit testing framework is not that important.
I'm looking for a completely automated tool, so that I can set it to run it after each build and instantly look at the results and compare them with previous results. The closest thing I've found is nunit-results and a hand-written batch file to call NUnit (with specified xml report path) and nunit-results as a post-build action. However, html file that it produces is not that informative.
I'm really surprised that noone of the popular unit testing software is capable of storing test run information and analyzing series of runs in bulk. I've tried Resharper, NUnit GUI, Gallio and haven't found anything useful.
I would be glad for a solution that does not require a setup of a complicated CI server. My projects are typically small, but I need a tool like this for every one of them.
I don't know what your threshold is for "complicated CI server", but Jenkins is pretty easy to setup, and with the NUnit Plugin ought to give you what you're after:
This plugin makes it possible to import NUnit reports from each build into Jenkins so they are displayed with a trend graph and details about which tests that failed.
If you are interested in a "Graph of progress", I'd go for a way more simple (IMHO) approach and use NCrunch. It shows you your tests status as you code, without stopping for test runs. See my answer here for more details.