We have managed to have Jenkins correctly parse our XML output from our tests and also included the error information, when there is one. So that it is possible to see, directly in the TestCase in Jenkins the error that occurred.
What we would like to do is to have Jenkins keep a log output, which is basically the console output, associated with each case. This would enable anyone to see the actual console output of each test case, failed or not.
I haven't seen a way to do this.
* EDIT *
Clarification - I want to be able to see the actual test output directly in the Jenkins interface, the same way it does when there is an error, but for the whole output. I don't want only Jenkins to keep the file as artifact.
* END OF EDIT *
Anyone can help us on this?
In the Publish JUnit test result report (Post-build Actions) tick the Retain long standard output/error checkbox.
If checked, any standard output or error from a test suite will be
retained in the test results after the build completes. (This refers
only to additional messages printed to console, not to a failure stack
trace.) Such output is always kept if the test failed, but by default
lengthy output from passing tests is truncated to save space. Check
this option if you need to see every log message from even passing
tests, but beware that Jenkins's memory consumption can substantially
increase as a result, even if you never look at the test results!
This is simple to do - just ensure that the output file is included in the list of artifacts for that job and it will be archived according to the configuration for that job.
Not sure if you have solve it yet, but I just did something similar using Android and Jenkins.
What I did was using the http://code.google.com/p/the-missing-android-xml-junit-test-runner/ to run the tests in the Android emulator. This will create the necessary JUnit formatted XML files, on the emulator file system.
Afterwards, simply use 'adb pull' to copy the file over, and configure the Jenkin to parse the results. You can also artifact the XML files if necessary.
If you simply want to display the content of the result in the log, you can use 'Execute Shell' command to print it out to the console, where it will be captured in the log file.
Since Jenkins 1.386 there was a change mentioned to Retain long standard output/error in each build configuration. So you just have to check the checkbox in the post-build actions.
http://hudson-ci.org/changelog.html#v1.386
When using a declarative pipeline, you can do it like so:
junit testResults: '**/build/test-results/*/*.xml', keepLongStdio: true
See the documentation:
If checked, the default behavior of failing a build on missing test result files or empty test results is changed to not affect the status of the build. Please note that this setting make it harder to spot misconfigured jobs or build failures where the test tool does not exit with an error code when not producing test report files.
Related
Is there a way to incorporate a clean up script in Postman?
Use case: After the collection run : (either success or failure). I need to clear data in some of the databases/data-stores
similar construct to try{} finally{}
for eg : as a part of collection runner contains two apis
api1 -> which puts the data in redis.
api2 -> functional verification
(expecting the clean up hook) to remove the data from that was put in step 1.
writing at the end of test script of api2 will work fine only if there are no errors in execution of test script.
the problem gets worse when there are large number of apis and multiple data entries. We can handle this by setNextRequest, however that brings additional code to be written in each test script.
You could probably achieve this by running the collection file within a script, using Newman. This should give you more flexibility and control over running certain actions at different points before, during and after the run.
More information about the different options can be found here: https://github.com/postmanlabs/newman/blob/develop/README.md#api-reference
If its just clearing out certian variable values, this can be done within the Tests tab of the last request in your collection.
Usually certain Grunt tasks have the option to "fail on warning", making it possible to put it as a step in your build process.
I am looking at JSBeautifier but it seems like there is no option.
Any suggestions on how we can make our build fail, if a JavaScript file gets formatted by JSBeautifier a build step?
If mode is "VERIFY_ONLY", then task will fail if at least one file can
be beautified. This is useful for pre-commit check.
https://www.npmjs.com/package/grunt-jsbeautifier#options-mode-optional
I would like to redirect the output of the Fabric command
print local("git add .")
so that I make sure that everything is working properly. Is there a way to pipe the results of this command to the console? Thanks!
You should already see the output of the command in the console since that's the default behaviour. If you need to customize it, please have a look at the managing output documentation section.
Besides this, there's an ongoing effort to use the logging module (issue #57) that should provide more options like logging to a file, but that hasn't been merged into the trunk branch yet.
I'm running my webApp using Jetty with my instrumented classes.
After the shutdown of Jetty i'm taking the generated .set file and creating a cobertura report using the command line tool.
I always get 100% coverage results on any class.
It seems that Cobertura takes into account only the lines that were executed during testing, and doesn't get the full class data.
I've tried to add source files to the reports - no help.
I also tried to take the .ser file created after the instrumentation and merge it with .ser file created after Jetty shutdown (it is actually the same file, but before running Jetty I backed-up the .ser which was created after instrumentation) - no help here either.
Can someone please help??
Thanks
100% coverage is a clear indicator, that the sources are missing for the report. You should check your configuration for creating the report.
Make sure that:
you give the right folder
the source folder is structured like the packages, and not just all classes in one dir
As explained at http://cobertura.sourceforge.net/faq.html, in the answer to the question "When I generate coverage reports, why do they always show 100% coverage everywhere?",
"Cobertura is probably using the wrong .ser file when generating the reports. When you instrument your classes, Cobertura generates a .ser file containing basic information about each class. As your tests run, Cobertura adds additional information to this same data file. If the instrumented classes can not find the data file when running then they will create a new one. It is important that you use the same cobertura.ser file when instrumenting, running, and generating reports."
In my case, I experienced this issue when instrumented classes were in one .ser and during execution I was generating another .ser. Generating the HTML report "just" from the second .ser shown the problem mentioned in the question. Merging the two datafiles (.ser), and regenerating the report, solved the issue.
Refer to http://cobertura.sourceforge.net/commandlinereference.html for "Merging Datafiles" information.
In CPP unit we run unit test as part of build as part of post build setup. We will be running multiple tests as part of this. In case if any test case fails post build should not stop, it should go ahead and run all the test cases and should report summary how many test cases passed and failed. how can we achieve this.
Thanks!
His question is specific enough. You need a test runner. Encapsulate each test in its own behavior and class. The test project is contained separately from the tested code. Afterwards just configure your XMLOutputter. You can find an excellent example of how to do this in the linux website. http://www.yolinux.com/TUTORIALS/CppUnit.html
We use this way to compile our test projects for our main projects and observe if everything is ok. Now it all becomes the work of maintaining your test code.
Your question is too vague for a precise answer. Usually, a unit test engine return a code to tell it has failed (like a non zero return code in the shell on linux) or generate some output file with results. The calling system handle this. If you have written it (some home made scripts) you have to give the option to go on tests execution even if an error occurred. If you are using some tools like continuous integration server, then you have to go through the doc and find the option that allows you to go on when tests fails.
A workaround is to write a script that return a "OK" result even if the unit test fails, but there you lose some automatic verification ...
Be more specific if you want more clues.
my2c
I would just write your tests this way. Instead of using the CPPUNIT_ASSERT macros or whatever you would write them in regular C++ with some way of logging errors.
You could use a macro for this too of course. Something like:
LOGASSERT( some_expression )
could be defined to execute some_expression and to log the expression together with FILE and LINE if it fails, and you can also log exceptions of course, as well as ones that are not thrown, simply by writing them in your tests (with macros if you want to log the expression that caused them with FILE and LINE).
If you are writing macros I would advise you to limit the content of your macro to calling an inline function with extra parameters.