There is an option in IDEA IntelliJ, that you can export test results (after running TestNG tests) in HTML or XML format. I am wondering, is it possible to generate them by default?
Take a look at the default TestNG logging reporters and what seems to be a cosmetically improved version, ReportNG. Depending on your build system, you can take advantage of what's already being provided, or you can supply your own implementation.
Related
Currently, I am using jacoco report tool to know how much coverage for my system. But this system is quite old and has many existing Fitnesse tests. I am newbie on this system and want to learn it by reading its Fitnesse test. My problem is I dont know what Fitnesse test is for what specific class.Beside that When I see a class is covered and I dont know which Fitnesse test covered this class. My system has more than 500 Fitnesse Tests.
JaCoCo does not provide this out of the box.
There are some third-party JaCoCo integrations such as SonarQube that allow to obtain information about relation between JUnit or TestNG test to code. Don't know much about Fitnesse, so can't tell whether this will work with it or not.
However referring to the same link: general principle to obtain such information using JaCoCo (and that's how SonarQube does) - is to measure coverage separately for each test and save data into place dedicated to this test.
Also IDEs are able to show coverage (there are EclEmma plugin for Eclipse based on JaCoCo and other tools for other IDEs), so if you're able to run Fitneese test in IDE, then this principle can be applied even manually by running one test after another within IDE.
Also have a look at another code coverage tool for Java that is commercial and named Clover.
I'm using plunit package for my prolog unit tests (SWI-Prolog 7.2).
run_tests/0 prints the results on console but I would like to export plunit test results in the xUnit XML format that most CI servers understand. Is there any way for this ?
I assume that SWI-Prolog plunit tool uses the message printing mechanism to generate its output. If true, you should be able to export unit test results in the xUnit XML format (or any other format) by intercepting those messages using the message_hook/3 predicate:
http://www.swi-prolog.org/pldoc/doc_for?object=message_hook/3
By coincidence, I'm working in similar support (for exporting testing results) for Logtalk's unit testing tool, lgtunit. It should give you an idea on how to do it for plunit. I committed a preliminary version today:
https://github.com/LogtalkDotOrg/logtalk3/blob/master/tools/lgtunit/NOTES.md
https://github.com/LogtalkDotOrg/logtalk3/blob/master/tools/lgtunit/xunit_xml_report.lgt
In my case, this support is being targeted for integration with the CI server Concourse. There seems to be, unfortunately, a lack of definitive information on the xUnit XML format with sources quoting different versions of e.g. which attributes are required or optional. I did find a XSD for this format bit I have no idea of its accuracy or if it's just another variation:
https://gist.github.com/erikd/4192748
How to control output from Twisted-trial tests?
I have looked up for different solutions, but I'm quite new to testing, so I can't find a fitting solution or can't use it correctly.
In general, I try to make autotesting system for my project like BuildBot. But BuildBot doesn't fit me because it reacts only to "On change sources" hook from Mercurial and I want to use other hooks too.
On THIS page from BuildBot documentation I found this information:
One advantage of trial is that the Buildbot happens to know how to
parse trial output, letting it identify which tests passed and which
ones failed. The Buildbot can then provide fine-grained reports about
how many tests have failed, when individual tests fail when they had
been passing previously, etc.
Does it mean that there is no way but to parse information from test-output?
Other possible solutions?
Besides, I looked up in Twisted documentation and found this class IReporter.
Is it a solution and if it is, how can I use it?
If it isn't, are there any other solutions?
P.S. Please, note, that tests have already been written, so I can only start them and can't modify source code.
You can format output from trial arbitrarily by writing a reporter plugin. You found the interface for that plugin already - IReporter.
Once you write such a plugin, you'll be able to use it by adding --reporter=yourplugin to your trial command line arguments.
You can see the list of reporter plugins that are already available on your system using trial --help-reporters. If you have python-subunit installed then you'll see subunit which is a machine-parseable format that might already satisfy your requirements. Unfortunately it's still a subunit v1 reporter and subunit v2 is better in a number of ways. Still, it might suffice.
I have a big mess with 100 tests in one class and running all of them by clicking "Test project (...). They run in a random order and I would like them to run in a specific order - from beginning to the end, the same order that I wrote them. In eclipse it's not a problem because eclipse just works like that, how to do it in netbeans?
Any help will be appreciated.
Edit (due to answers): Tests order is required for the clearance of the log. They are independent.
If your tests needs to run in a specific order, something is wrong with your design.
2 test that needs to run one after another are 1 test. Consider this before searching for a solution.
check this https://blogs.oracle.com/mindless/entry/controlling_the_order_of_junit
Having tests depending on other tests 99.9% of the time a very bad idea. Unit tests should be independent from each other, as otherwise you might have a cascade of errors, or (even worse) one test failing because something another test did sometime before.
If you still want to go through this pain, you'll need to use a different unit testing framework (such as TestNG - see dependsOnMethods) which supports test dependencies.
Junit doesn't support this feature because it's seen by many as a bad practice (for very good reasons).
The next JUnit release will support ordering of test methods. The standard Maven Surefire Plugin supports ordering of test methods already.
Netbeans has good integration with ant build files. You could write a specific ant target that could execute each test in order.
I googled and found the below helpful references. Currently I want to run all from the command-line (for easy of execution & quickness) in cases:
A specific test (ie. a test written by a method marked [TestMethod()])
All tests in a class
All impacted tests of the current TFS pending change of mine.
All tests
All tests except the ones marked as category [TestCategory("some-category")]
I'm not sure how can I write a correct command for my needs above.
References:
the MSTest.exe http://msdn.microsoft.com/en-us/library/ms182487.aspx
the MSTest.exe's detailed options http://msdn.microsoft.com/en-us/library/ms182489.aspx
obtaining the result http://msdn.microsoft.com/en-us/library/ms182488.aspx
[Edit]
After a while, I found the below useful tips.
run Visual Studio unit tests by using MSTest.exe, located at %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe in my case.
using /testcontainer:Path\To\Your\TestProjectAssembly.dll to indicate where your tests are coded. You can specify multiple '/testcontainer' options if required.
using /test:TestFilter to filter the tests to run. Note that this filter is applied to the full test method name (ie. FullNamespace.Classname.MethodName)
Currently I can have some answers for my needs:
A specific test (ie. a test written by a method marked [TestMethod()])
Use MSTest.exe /container:TheAssemblyContainingYourSpecificTest /test:TheSpecificTestName
All tests in a class
Use MSTest.exe /container:TheAssemblyContainingYourClass /test:TheClassNameWithFullNamespace
Note that the /test: is the filter which uses the full name of the class when filtering.
The others are still left unknown. Please disscuss if you know how.
For number 4. To run all tests in an assembly it's simply:
mstest /testcontainer:YourCompiledTestAssembly.dll
For question
5 All tests except the ones marked as category
[TestCategory("some-category")]
Use
mstest.exe /testcontainer:yourTests.dll /category:"!some-category"
If you need to exclude more than one category, use
mstest.exe /testcontainer:yourTests.dll /category:"!group1&!group2"
Reference: /category filter
You might be interested by the Gallio bundle. It provides a free common automation platform to run your tests (MSTest, MbUnit, NUnit, xUnit, etc.) with various test runners (GUI, command line, PoSh, plugins for 3rd party tools, etc.)
In particular you may want to use Gallio.Echo which is a nice command line test runner:
The Gallio test runners have also filtering capabilities to run a subset of your unit tests only (e.g. per category, per fixture, etc.)
** adding this due to errors I've encountered.
To run all just use '''vstest.console.exe .\x64\Release\UnitTesting.dll'''
vstest.console.exe is not deprecated so you will not need the /nologo suppression.
If needed it also has --TestCaseFilter|/TestCaseFilter: