I am working with Selenium tests in python using Nose as a test runner. I run my tests like so
nosetests -a level=gold --with-id --with-xunit
Once tests are done, I usually run
nosetests --failed
So far we have run tests using FireFox and Chrome webdrivers with no issues. It is not uncommon for one or two tests to fail (as our website is undergoes frequent builds, which causes tests to briefly fail), and only these tests are retried.
When I use PhantomJS's Ghostdriver, behavior is similar to Chrome/FF as one or two tests fail. But when I run nosetests --failed ALL tests are rerun, not just the failed ones.
webdriver setup a such:
self.driver = webdriver.PhantomJS(executable_path='C:\\SeleniumTests\\phantomjs.exe')
nosetest.xml on the first pass with phantomjs outputs
<?xml version="1.0" encoding="UTF-8"?><testsuite name="nosetests" tests="37" errors="1" failures="0" skip="0">
but on the second pass all 37 tests are rerun.
Is this a known issue with Ghostdriver? Or is there some something I am missing?
Related
I'm attempting to set up a new django project, and I've configured TEST_RUNNER in settings.py to be django_nose.NoseTestSuiteRunner.
I chose this test runner because it seems to be the only one I can find that has the following features:
writes xunit xml test report
captures logging/stdout and only displays for failed tests.
However I've heard that nose is unmaintained and I'm having a hard time finding a suitable replacement. The standard test runner doesn't capture logging nor writes xunit as far as I'm able to tell (would love to be proven wrong!)
I run tests like so:
python -m coverage run manage.py test --noinput
python -m coverage report --include="app/*" --show-missing --fail-under=100
python -m coverage xml --include="app/*" -o ./reports/coverage.xml
With this in settings.py:
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
And this setup.cfg:
[nosetests]
verbosity=0
with-xunit=1
xunit-file=./reports/xunit.xml
logging-clear-handlers=1
The last two lines are the real juicy bits I can't seem to find in other test runners. nose captures the logging and clears other logging handlers (eg, the handler that dumps on stdout) so the test runs output is much cleaner (you only see logging for tests that failed).
In other non-django projects I typically use nose2 but django-nose2 project appears to be 6 years old and lacking python3 support??
Please let me know which test runner is the "recommended" one (eg, most popular) with django support, thanks.
I have had success with unittest-xml-reporting:
TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner'
https://github.com/xmlrunner/unittest-xml-reporting#django-support
The output directory can be configured with the TEST_OUTPUT_DIR setting.
You may still use nose runner:
INSTALLED_APPS += ['django_nose']
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
NOSE_ARGS = [
'--with-xunit',
'--xunit-file=nosetests.xml',
'--with-coverage',
'--cover-erase',
'--cover-xml',
'--cover-xml-file=nosecover.xml',
]
So pytest produces some very nice test output. I've unset TEST_RUNNER in settings.py and changed my test script to:
python -m coverage run -m pytest --junitxml=./reports/junit.xml
python -m coverage report --include="app/*" --show-missing --fail-under=100
python -m coverage xml --include="app/*" -o ./reports/coverage.xml
This works, and captures ALL logging output (nose was a little buggy and let one or two logging statements slip through, very strange behavior).
The only thing is that I'm a django novice so I don't know if there are any bad side-effects of not using manage.py test for testing django. Any guidance is appreciated, thanks!
I've installed nose and coverage to my virtual env, but it doesn't work
(venv) ../my_cookbook$ nosetests --with-coverage
nose.plugins.cover: ERROR: Coverage not available: unable to import coverage module
I wondered if it was escaping my venv somehow, so I tried this and it worked!
(venv) ../my_cookbook$ ./venv/bin/nosetests --with-coverage
Then I wanted to see if my path was some how messed up.
(venv) ../my_cookbook$ which nosetests
/home/peter/Projects/my_cookbook/venv/bin/nosetests
(venv) ../my_cookbook$ which coverage
/home/peter/Projects/my_cookbook/venv/bin/coverage
So what is going on here? Somehow the nosetests command is escaping my virtualenv but I don't know how.
Unfortunately, I don't have an explanation why the nose plugin is not picking up coverage, but, executing your tests through coverage should be preferred as opposed to using test runner plugins (nose coverage plugin in your case). Quoting Ned Batchelder (the author of coverage):
using a plugin means you are depending on that plugin's behavior being correct and understandable. In the name of being helpful, plugins will have their own logic that may have been the best idea when they were written, but the test runner and/or coverage.py may have changed in the meantime. The plugins tend not to be as well-maintained as the other components. If you can avoid them, you have one less thing to think about.
In other words, run:
$ coverage run -m nose
and to get the coverage report:
$ coverage report
I have successfully converted two projects from grails 2.2.4 to 2.4.3. I can run the tests, and run-app both from the grails command line, and within ggts.
Then, I converted it to a maven project, by using create-pom to get an initial pom.xml file. After getting the dependencies right, I can again run the tests and the app from the command line and ggts. But I am getting errors on my unit tests (not spock tests) using 'mvn test'.
The errors look like this:
java.lang.NullPointerException
at grails.test.runtime.GrailsApplicationTestPlugin.createParentContext(GrailsApplicationTestPlugin.groovy:143)
at grails.test.runtime.GrailsApplicationTestPlugin.initGrailsApplication(GrailsApplicationTestPlugin.groovy:96)
at grails.test.runtime.GrailsApplicationTestPlugin.onTestEvent(GrailsApplicationTestPlugin.groovy:327)
at grails.test.runtime.TestRuntime.deliverEvent(TestRuntime.groovy:295)
It feels like some kind of setup , or dependency issue.
I'm in the middle of my first ever stab at setting up Jenkins to build and run unit tests /code coverage with my CakePHP project. So far I have successfully got Jenkins fetching and building automatically from my BitBucket repository - a small victory in itself.
Next thing I want to happen is for the unit tests to run and code coverage reports to be populated.
Here is my build.xml, which is being executed in Jenkins with the (only) build command phing -f $WORKSPACE/build.xml
<?xml version="1.0" encoding="UTF-8"?>
<project name="Consumer Love" default="phpunit">
<target name="phpunit">
<exec command="cake test app --coverage-clover logs/reports/clover.xml"></exec>
</target>
</project>
I think the issue is that when you run cake test app it asks for a prompt of which specific tests you want to run, I have been unable to figure out a method to run all of my CakePHP app unit tests.
The solution was to create a custom CakePHP Test suite which adds specific files/directories to be tested, then run that suite with the command cake test app AllTests.
For example, here is my Test/Case/AllTests.php:
/*
* Custom test suite to execute all tests
*/
class AllTestsTest extends PHPUnit_Framework_TestSuite {
public static function suite() {
$path = APP . 'Test' . DS . 'Case' . DS;
$suite = new CakeTestSuite('All tests');
$suite->addTestDirectory($path . 'Model' . DS);
return $suite;
}
}
This testsuite simply adds the Models directory to the testing environment, so all my model tests now get executed. As you can see it can be extended to run more/all tests as seen fit.
Try cake test app all. I can't confirm this makes the difference just now, but I've pulled this out of a phing build file where I'm doing the same thing as you so it should be good.
It is possible to run Scala's partest test suite with something like ant test, but is it possible to just rerun a single, failed test?
Additionally, is it possible to run only tests which have failed in a test run before?
If you are referring to the Scala repository build and the partest test suite, after having run the default ant build task, you can either run:
test/partest --failed
to run the last set of tests that failed, or:
test/partest test/files/<folder>/<testname>
This assumes you have previously done a:
test/partest --all
or called partest for some other test category.