boost unit tests with xUnit Plugin in Jenkins do not work - unit-testing

I am not a C programmer, but i have to run boost tests on my Jenkins. Now I have installed the xUnit plugin in Jenkins.
I added a post-build action : "Publish xUnit test result report"
Then, in this post-build step I added : "BoostTest-1.x (default)"
Now I have the following options to set:
https://www.dropbox.com/s/wxcny55rz2bqk6r/boost_jenkins_options.png
The options I set are random, so please help me, I don't understand anything and I didn't find some tutorials.
I have not worked with boost unit test and not with the xUnit Jenkins plugin either.
Can any one help me?
edit: jenkins say me this:
make[1]: Leaving directory `/var/lib/jenkins/workspace/southernd_test'
[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing BoostTest-1.x (default)
[xUnit] [INFO] - [BoostTest-1.x (default)] - No test report file(s) were found with the pattern 'boost/*.xsl' relative to '/var/lib/jenkins/workspace/southernd_test' for the testing framework 'BoostTest-1.x (default)'. Did you enter a pattern relative to the correct directory? Did you generate the result report(s) for 'BoostTest-1.x (default)'?
[xUnit] [ERROR] - No test reports found for the metric 'BoostTest' with the resolved pattern 'boost/*.xsl'. Configuration error?.
[xUnit] [INFO] - Setting the build status to FAILURE
[xUnit] [INFO] - Stopping recording.
Build step 'Publish xUnit test result report' changed build result to FAILURE
Finished: FAILURE

The error is is because there is no output file generated by boost::test. The test script need to be invoked with the correct options:
unit_test --report_level=detailed --report_format=xml 2> xunit.xml
Unfortunately the XML output file produced by boost::test is not in the correct format
(see: SO Converting boost::test logs & Boost Users Help with XUnit plugin )
The JUnit plugin expects XML test output to be in the following format:
<testsuites>
<testsuite time="0.0000" timestamp="0.000" errors="0" failures="0" tests="13" hostname="localhost" name="my_test_suite">
<testcase id="65536" class="test" name="test_case_1" time="0.0000" />
<testcase id="65537" class="test" name="test_case_2" time="0.0000" />
<testcase id="65538" class="test" name="test_case_3" time="0.0000" />
</testsuite>
</testsuites>
There are a couple of ways to resolve this such as:
Converting the XML output by boost::test
Directly customising the output of boost::test so that the correct format is produced.
I opted for option 2 - ss you are not a 'C/C++' programmer you could get the author of the test suites you are trying to run to follow this approach, the steps below should help get them started:
Create a test visitor for post processing the results of the test run.
Create a BOOST_GLOBAL_FIXTURE class that walks through the test results in its destructor to output the test results in the correct format.
Instantiate the fixture class in the main test module.
i.e.:
struct JUnitVisitor : public boost::unit_test::test_tree_visitor
{
void visit( boost::unit_test::test_case const& tc )
{
// output <testcase> xml in JUnit format
}
bool test_suite_start( boost::unit_test::test_suite const& ts )
{
// output <testuite> xml in JUnit format
}
void test_suite_finish( boost::unit_test::test_suite const& ts )
{
// output </testuite> xml in JUnit format
}
};
struct MyJUnitOpFixture
{
MyJUnitOpFixture() {}
~MyJUnitOpFixture()
{
// open results file
/// output <testsuites> start tag
// use a visitor to walk the test results tree
JUnitVisitor visitor ( out );
boost::unit_test::traverse_test_tree(
boost::unit_test::framework::master_test_suite(),
visitor
);
/// output </testsuites> end tag
}
}
Then the global fixture is instantiated in the main test file by adding:
BOOST_GLOBAL_FIXTURE( MyJUnitOpFixture );

In my case xUnit does not like format of Boost Test's "--report_format=XML" but it DOES take "--log_format=XML --log_sink=test.xml"

Related

phpunit replies unrecongnized option --env

I was trying to execute my test cases using phpunit command in laravel but the command replies the following
$ phpunit --env=testing
PHPUnit 3.7.28 by Sebastian Bergmann.
unrecognized option --env
Is this familiar to someone please help me.
Edit:
The following is the help for the command where the option does not exist. What should I do?
$ phpunit --help
PHPUnit 3.7.28 by Sebastian Bergmann.
Usage: phpunit [switches] UnitTest [UnitTest.php]
phpunit [switches] <directory>
--log-junit <file> Log test execution in JUnit XML format to file.
--log-tap <file> Log test execution in TAP format to file.
--log-json <file> Log test execution in JSON format.
--coverage-clover <file> Generate code coverage report in Clover XML format.
--coverage-html <dir> Generate code coverage report in HTML format.
--coverage-php <file> Serialize PHP_CodeCoverage object to file.
--coverage-text=<file> Generate code coverage report in text format.
Default to writing to the standard output.
--testdox-html <file> Write agile documentation in HTML format to file.
--testdox-text <file> Write agile documentation in Text format to file.
--filter <pattern> Filter which tests to run.
--testsuite <pattern> Filter which testsuite to run.
--group ... Only runs tests from the specified group(s).
--exclude-group ... Exclude tests from the specified group(s).
--list-groups List available test groups.
--test-suffix ... Only search for test in files with specified
suffix(es). Default: Test.php,.phpt
--loader <loader> TestSuiteLoader implementation to use.
--printer <printer> TestSuiteListener implementation to use.
--repeat <times> Runs the test(s) repeatedly.
--tap Report test execution progress in TAP format.
--testdox Report test execution progress in TestDox format.
--colors Use colors in output.
--stderr Write to STDERR instead of STDOUT.
--stop-on-error Stop execution upon first error.
--stop-on-failure Stop execution upon first error or failure.
--stop-on-skipped Stop execution upon first skipped test.
--stop-on-incomplete Stop execution upon first incomplete test.
--strict Run tests in strict mode.
-v|--verbose Output more verbose information.
--debug Display debugging information during test execution.
--process-isolation Run each test in a separate PHP process.
--no-globals-backup Do not backup and restore $GLOBALS for each test.
--static-backup Backup and restore static attributes for each test.
--bootstrap <file> A "bootstrap" PHP file that is run before the tests.
-c|--configuration <file> Read configuration from XML file.
--no-configuration Ignore default configuration file (phpunit.xml).
--include-path <path(s)> Prepend PHP's include_path with given path(s).
-d key[=value] Sets a php.ini value.
-h|--help Prints this usage information.
--version Prints the version and exits.
You should know your PHPUnit version is VERY outdated, Nov 2014. Please update.
Also, what are you trying to do with the --env? You can configure the application environment in the phpunit.xml in the root directory like so:
<php>
<env name="APP_ENV" value="testing"/>
... more variables
</php>

Junit5 does not report skipped tests

I am working on a java pluging which is built from a Jenkins project, the build is done using gradle.
From times to times, we may have to skip a test method, or a test overall. I do that by using the #Disable annotation.
However, in this case, the test method, or test, simply disappear from the xml report. I expect the test to be reported as "Skipped" in the xml report so Jenkins can know it was skipped and display it in yellow.
Any solution to that ?
If you are using the junitPlatform() support in Gradle, Gradle will generate XML output similar to the following showing that a test has been skipped.
<testcase name="enabledAndDisabled()" classname="org.junit.jupiter.extensions.EnabledIfTests" time="0.0">
<skipped/>
</testcase>
I copied the above from the TEST-org.junit.jupiter.extensions.EnabledIfTests.xml generated in the platform-tests module of the JUnit 5 build.

How to configure Gitlab CI to parse test output?

I am working on Gitlab and I would like to set up a CI (it is the first time I configure something like that, please assume that I am a beginner)
I wrote a code in C with a simple test in Cunit, I configured CI with a "build" job and a "test" job. My test job succeed whereas I wrote a KO test, when I open the job on Gitlab I see the failed output, but the job is marked "Passed".
How can I configure Gitlab to understand that the test failed ?
I think there is a parsing configuration somewhere, I tried in "CI / CD Setting -> Test coverage parsing" but I think it is wrong, and it did not work.
I let you the output of my test :
CUnit - A unit testing framework for C - Version 2.1-2
http://cunit.sourceforge.net/
Suite: TEST SUITE FUNCTION
Test: Test of function::triple ...FAILED
1. main.c:61 - CU_ASSERT_EQUAL(triple(3),1)
Run Summary: Type Total Ran Passed Failed Inactive<br/>
suites 1 1 n/a 0 0<br/>
tests 1 1 0 1 0<br/>
asserts 3 3 2 1 n/a<br/>
Elapsed time = 0.000 seconds
Gitlab supports test reports in JUnit format and coverage reports in cobertura XML format
See the links for C++ examples that may help you, as an example for CUnit, the .gitlab_ci.yaml file should look like:
cunit:
stage: test
script:
- ./my-cunit-test
artifacts:
when: always
reports:
junit: ./my-cunit-test.xml

How to extract readable format report from jacoco.exec?

I have a jacoco-agent generated file of my Maven project (Java), named jacoco.exec.
How can I convert this file into human readable format? (HTML/XML).
I believe that this is described in official JaCoCo documentation. In particular there is jacoco-maven-plugin goal "report" and example of its usage.
Starting from JaCoCo version 0.8.0 there is also Command Line interface. Here is an example of how to use it to produce HTML report from jacoco.exec, note that this also requires class files:
java -jar jacoco-0.8.1/lib/jacococli.jar report jacoco.exec --classfiles directory_with_classes --html directory_for_report
In the Gradle world (for anyone who happened to stumble upon this question), use the JacocoReport task, documented here.
The configured task would look something like this:
jacocoTestReport {
reports {
html.enabled = true
html.destination "${buildDirectory}/coverage/report/html"
xml.enabled = true
xml.destination "${buildDirectory}/coverage/report/xml"
}
executionData = files("path/to/jacoco.exec")
}
Godin's answer is perfect. But if you are using maven, then you can simply run the following command
mvn clean verify
As Jacoco's report goal is attached to the verify phase of maven life cycle.
mvn verify would simply invoke the report goal, and will generate all the reports in the following path
./target/site/jacoco

Problem running tests under Netbeans

I am running some Junit tests and Netbeans behaves strangely giving the report in "Output" window:
Testcase: warning(junit.framework.TestSuite$1): FAILED
No tests found in uk.ac.cam.ch.wwmm.chemicaltagger.ChemistryPOSTaggerTest
junit.framework.AssertionFailedError: No tests found in uk.ac.cam.ch.wwmm.chemicaltagger.ChemistryPOSTaggerTest
Test uk.ac.cam.ch.wwmm.chemicaltagger.ChemistryPOSTaggerTest FAILED (crashed)
test:
BUILD SUCCESSFUL (total time: 12 seconds)
The (5) tests are there. I have run mvn test which runs them but fails on OutOfMemoryError. Is this likely to be the cause of the Netbeans problem?
How did you created your test file? Manually, or using NB wizard? (Tools - Create JUnit test from Java file's popup menu)
If you are using JUnit 3, all test methods in your test file must start with "test", e.g.
public void testFoo() { //some testing here :) }
With JUnit 4, a '#Test' annotation is required, e.g.
#Test
public void myOwnTestFoo() { //...}
Otherwise JUnit does not recognize the test and throws AssertionFailedError error.