I am new to testing in Xcode. I am following the developer documentation "iOS Developement Guide" Unit Testing Applications.
I have successfully added unit test bundles to my application with one failing test.
When I build the test target the output shows 1 error as expected but the editor does not show the expected error message under the failed test.
Does anyone know any configurations that need to be set to enable this feature?
Apples docs say:
"If the unit-test bundle is configured correctly, the build fails and Xcode displays an error message in the text editor."
TIA
Vital Clue:
- One thing I have noticed in my output window is the message "Command /bin/sh failed with exit code 1"
Try to follow this guide
Be double careful about when to set application target, and when to set Unit test target. It works fine for me though.
If Unit Test fail, there should be some messages shown in the console.
Related
Currently the only thing I can see is the executable exit code, so I can see if any tests failed or not, but I don't get any other output.
Is there a way to show something similar to what you see in Qt Creator when running tests?
I am using CMake and the Qt Test framework.
You can use -junitxml option to output test results to xml file(s) and use one of actions to watch detailed reports:
action-junit-report
publish-unit-test-results
junit-report-action
test-reporter
report-junit-annotations-as-github-actions-annotations
I figured out a workaround. If you use the -o filename,format command line argument and output to a file, you can then use more filename to get the Test results.
GitHub actions uses Windows Server as the environment so that might be a reason for the weird behavior. My best guess is the Qt implementation works differently on Windows Server, because std::out works fine.
We are using IBM Rational Test RealTime for unit testing of ANSI-C code. It's my job to tell a Jenkins server to run those tests automatically.
Documentation suggests what command line should be used:
studio -r <node>.{[.<node>]} <project_file> [-html <directory>]
Unfortunately running studio -r with any more parameter does not output anything to the console. If it does not do the intended unit testing job, it terminates immediately and gives no error message whatsoever. This is frustrating. The error class can be deduced from the exit code, but that is not really enough for troubleshooting.
Any idea where to find error output? I can not find any log file.
The detailed command line syntax is described here,
but this claims, what apparently is not true:
All messages are sent to the standard error output device.
If it runs successfully missing console output can be tollerated, but if it fails to run the test, I really, really would like to see the stuff that is normally output into the RTRT main window. Note: a unit test that fails
e.g. on an assertion is counted as "successful" test run in this context.
I am running tests through Jenkins on a windows box. In my "Execute Windows Batch command" portion of the project configuration I have the following command:
nosetests --nocapture --with-xunitmp --eval-attr "%APPLICATION% and priority<=%PRIORITY% and smoketest and not dev" --processes=4 --process-timeout=2000
The post build actions have "Publish JUnit test result report" with the Test report XMLs path being:
trunk\automation\selenium\src\nosetests.xml
When I do a test run, the nosetests.xml file is created, however it is empty, and I am not getting any Test Results for the build.
I am not really sure what is wrong here.
EDIT 1
I ran the tests with just --with-xunit and REM'd out the --processes and got test results. Does anyone of problems with xunitmp not working with a Windows environment?
EDIT 2
I unstalled an reinstalled nose and nose_xunitmp to no avail.
The nosetest plugin for parallelizing tests and plugin for producing xml output are incompatible. Enabling them at the same time will produce the exact result you got.
If you want to keep using nosetest, you need to execute tests sequentially or find other means of parallelizing them (e.g. by executing multiple parallel nosetest commands (which is what I do at work.))
Alternatively you can use another test runner like nose2 or py.test which do not have this limitation.
Apparently the problem is indeed Windows and how it handles threads. We attempted several tests outside of our Windows Jenkins server and they do not work either. Stupid Windows.
Currently I am building and testing my app in Jenkins. If a test fails in the job then the build is marked as Unstable. Is it possible to have the build be marked as Stable even though tests fail? The build is marked as Failed if code does not compile, and that is fine.
There is a plugin called Jenkins Text Finder:
https://wiki.jenkins-ci.org/display/JENKINS/Text-finder+Plugin
It allows to search console log for regular expression matches, and if found, it can set the build to various states, including setting it as "success" (i.e. not unstable).
Just echo a unique label/line into your console output after the build is successful, and look for that line with this plugin. Use "Succeed if found" option in the plugin to mark build successful.
I have a strange problem,
my MSBuild runs tests, code-coverage and publishing fine (part of the build.txt shown):
Results Top Level Tests
------- ---------------
Passed BuildTestProject.UnitTest1.TestMethod1
Passed BuildTestProject.UnitTest1.TestMethod2
2/2 test(s) Passed
...
Results file: W:\BuildWorkspace\XXX\Test Release\TestResults\XXX_XXX 2009-08-20 11_47_09_Any CPU_Release.trx
Run Configuration: Local Test Run
Waiting to publish...
Publishing results of test run XXX#XXX 2009-08-20 11:47:09_Any CPU_Release to http://XXX:8080/Build/v1.0/PublishTestResultsBuildService2.asmx...
....Publish completed successfully.
When I import these testresults on my local machine i get to see the code coverage-data as expected. But the code-coverage details are not shown in the details of the build that Visual Studio shows when you expand the 'results details'.
Any tips?
Finally found the solution today, found out that my TeamFoundationServer itself had problems with publishing the results to my buildAgent. I read somewhere (after searching for CoverAn.exe) that it is installed as a service, then i checked the credentials for the service 'Code Coverage Analysis Service'.
These were running under TFSMachine\NETWORK instead of our service-account for the TeamFoundationServer. Changed this, reran a build with tests in it and it now publishes the results.
W00t!
Check that the .testrunconfig used by the server has coverage enabled. It may be different than your local .testrunconfig.