Simply put:
When I execute the tests using boost runner I get the following error and the c/c++ unit testing hangs.
XML parse error: The entity name must immediately follow the '' in the entity reference.
How do I avoid this from happening?
FYI:
The test runners run perfectly fine from the command line outside of eclipse, or in a debug window in eclipse, so it must be an eclipse issue isolated in the "c/c++ unit console".
Im running Kepler.
Gracias.
If you put a BOOST_CHECK inside a test stub, and the tested code is multithreaded, the asynchronous <Info> output will probably corrupt the XML output. Check the XML output carefully to spot what is the offending test.
At least up to 1.57, while validating stub usage is also important for unit-testing modules, Boost.Test is not considered thread-safe in this context.
You could probably work around the problem by redirecting the output stream and overloading boost::unit_test::unit_test_log.set_stream.
Related
Currently the only thing I can see is the executable exit code, so I can see if any tests failed or not, but I don't get any other output.
Is there a way to show something similar to what you see in Qt Creator when running tests?
I am using CMake and the Qt Test framework.
You can use -junitxml option to output test results to xml file(s) and use one of actions to watch detailed reports:
action-junit-report
publish-unit-test-results
junit-report-action
test-reporter
report-junit-annotations-as-github-actions-annotations
I figured out a workaround. If you use the -o filename,format command line argument and output to a file, you can then use more filename to get the Test results.
GitHub actions uses Windows Server as the environment so that might be a reason for the weird behavior. My best guess is the Qt implementation works differently on Windows Server, because std::out works fine.
We are using IBM Rational Test RealTime for unit testing of ANSI-C code. It's my job to tell a Jenkins server to run those tests automatically.
Documentation suggests what command line should be used:
studio -r <node>.{[.<node>]} <project_file> [-html <directory>]
Unfortunately running studio -r with any more parameter does not output anything to the console. If it does not do the intended unit testing job, it terminates immediately and gives no error message whatsoever. This is frustrating. The error class can be deduced from the exit code, but that is not really enough for troubleshooting.
Any idea where to find error output? I can not find any log file.
The detailed command line syntax is described here,
but this claims, what apparently is not true:
All messages are sent to the standard error output device.
If it runs successfully missing console output can be tollerated, but if it fails to run the test, I really, really would like to see the stuff that is normally output into the RTRT main window. Note: a unit test that fails
e.g. on an assertion is counted as "successful" test run in this context.
I have a C++ project in NetBeans using generated Makefiles. I set up a job in Jenkins (continuous integration server) to run the tests configured in NetBeans. Now Jenkins runs the tests and captures their output, but it considers the build successful even when a test fails.
I'm using the Boost Unit Test Framework which of course returns a non-zero code on failure as any proper *nix program would. So I wondered why Jenkins didn't understand when a test failed. Then I found this in the generated Makefile-Debug.mk from NetBeans:
# Run Test Targets
.test-conf:
#if [ "${TEST}" = "" ]; \
then \
${TESTDIR}/TestFiles/f1 || true; \
${TESTDIR}/TestFiles/f2 || true; \
else \
./${TEST} || true; \
fi
So it seems like they deliberately ignore the return value of all tests. But this doesn't make sense, because then what are your tests testing?
I tried to find a setting in NetBeans to say "Let failing tests break the build" but didn't find anything. I also tried to find a bug in the NetBeans tracker for this but didn't see any in my brief search.
Is there any other reasonable solution? I want Jenkins to fail my build if any test fails. Right now it only fails if a test fails to build, but if it builds and fails to run, success is reported.
It turns out that NetBeans (up to version 8 at least) cannot support this. What I did to work around it is to do make build-tests rather than make test in Jenkins, followed by a loop over all the generated test files (TestFiles/f* in the build directory) to run them.
This is a major shortcoming in NetBeans' Makefile generator, as it is fundamentally incompatible with running tests outside of NetBeans itself. Thanks to #HEKTO for the link which led me to this page about writing NetBeans testing plugins: http://wiki.netbeans.org/CND69UnitTestsPluginTutotial
What that page tells you is basically that NetBeans relies on parsing the textual output of tests to determine success or failure. What it doesn't tell you is that NetBeans generates defective Makefiles which ignore critical failures in tests, including aborts, segmentation faults, assertion failures, uncaught exceptions, etc. It assumes you will use a test framework that it knows about (which is only CppUnit), or manually write magic strings at the right moments in your test programs.
I thought about taking the time to write a NetBeans unit test plugin for the Boost Unit Test Framework, but it won't help Jenkins at all: the plugins are only used when tests are run inside NetBeans itself, to display pretty status indicators.
I am running tests through Jenkins on a windows box. In my "Execute Windows Batch command" portion of the project configuration I have the following command:
nosetests --nocapture --with-xunitmp --eval-attr "%APPLICATION% and priority<=%PRIORITY% and smoketest and not dev" --processes=4 --process-timeout=2000
The post build actions have "Publish JUnit test result report" with the Test report XMLs path being:
trunk\automation\selenium\src\nosetests.xml
When I do a test run, the nosetests.xml file is created, however it is empty, and I am not getting any Test Results for the build.
I am not really sure what is wrong here.
EDIT 1
I ran the tests with just --with-xunit and REM'd out the --processes and got test results. Does anyone of problems with xunitmp not working with a Windows environment?
EDIT 2
I unstalled an reinstalled nose and nose_xunitmp to no avail.
The nosetest plugin for parallelizing tests and plugin for producing xml output are incompatible. Enabling them at the same time will produce the exact result you got.
If you want to keep using nosetest, you need to execute tests sequentially or find other means of parallelizing them (e.g. by executing multiple parallel nosetest commands (which is what I do at work.))
Alternatively you can use another test runner like nose2 or py.test which do not have this limitation.
Apparently the problem is indeed Windows and how it handles threads. We attempted several tests outside of our Windows Jenkins server and they do not work either. Stupid Windows.
In my project I write tests using Microsoft's unit testing framework. All of my tests pass when I run them from Visual Studio but when I run the tests from MSBuild all of the tests fail with the following erorr message:
Unit Test Adapter threw exception:
Type is not resolved for member
SomeType,SomeAssembly Version=assemblyVersion,
Culture=neutral, PublicKeyToken=..
The assembly not found is a 3rd party assembly referenced by all of the projects.
The build script is used by TFS so I've aded the following lines:
<RunTest>true</RunTest>
<ItemGroup>
<MetaDataFile Include="$(BuildProjectFolderPath)myproject.vsmdi">
<TestList>CI_Tests</TestList>
</MetaDataFile>
</ItemGroup>
I've found the this post that shows a solution to this issue but unfortunatly I cannot chnage the files on the TFS server.
Help!
I encountered the same problem in my unit tests. The linked article above indicates that the problem is that VSTS causes copying of some objects in the thread's CallContext.
For what it's worth, in my case the problem was that I had manually placed an object in the thread's CallContext, which caused this exception. I was able to resolve this by clearing the CallContext in my TestCleanup routine. I didn't have to change any files anywhere.
I had also run into the same issue but where I had StructureMap initialisation being performed within the constructor for a base test class.
I was able to get around the problem by moving the call from the constructor to the [TestInitialize] method. I also ensured that the [TestCleanUp] method disposed of the created StructureMap container. After this MSBuild (2010) would run through the tests without raising this error.
The first thing to check would be if this assembly is copied to the folder from which msbuild runs the tests. It might be the case that you have a copy in your bin/Debug folder because of some historic reasons, but the dependency is not set up properly in the project.
Had this error
Unit Test Adapter threw exception:
Type is not resolved for member 'NHibernate.HibernateException,NHibernate
As it turned out the problem was in exception thrown in static constructor for the test.
It was completely unrelated to the looks of message and was happening during DB creation using BuildSchema.
Very uninformative error message by MSTest which cost me a lot of hours and stress. putting migration to something better like NUnit in our TODO list.
This article solved my problem with this error:
To recap, MyCustomException was thrown at very early stage of test execution. The dll that contained it was not loaded yet so Unit Test Adapter indeed could not reach it.
Happening when the assemblies been tested have loaded instances of their class marked as [Serializable]. Workaround by deriving from MarshalByRefObject so the deserialization is no longer attempted from mstest host.