Why does a failing test with mox fail other tests as well? - unit-testing

Question

My issue is quite simple: I'm having a bunch of unit tests using pymox. When I add a new test that fails, most of the time a whole lot of others fail as well. How can I prevent that from happening?
For example, I have a simple script for which I have two unit tests:
def test_main_returnsUnknown_ifCalculator_returnsMinus1(self):
m=mox.Mox()
m.StubOutWithMock(check_es_insert,"getArgs")
check_es_insert.getArgs(\
'Nagios plugin for checking the total number of documents stored in Elasticsearch')\
.AndReturn({ 'critical' : 7, 'warning' : 5, 'address' : 'myhost:1234', 'file' : '/tmp/bla'})
################
#some other mocking here, not relevant, I think
################
m.ReplayAll()
#now let's test
check_es_docs.main()
#verify and cleanup
m.UnsetStubs()
m.VerifyAll()
m.ResetAll()
def test_main_doesWhatPrintAndExitSays_inNormalConditions(self):
m=mox.Mox()
m.StubOutWithMock(check_es_insert,"getArgs")
check_es_insert.getArgs(\
'Nagios plugin for checking the total number of documents stored in Elasticsearch')\
.AndReturn({ 'critical' : 7, 'warning' : 5, 'address' : 'myhost:1234', 'file' : '/tmp/bla'})
################
#some other mocking here, not relevant, I think
################
m.ReplayAll()
#now let's test
check_es_docs.main()
#verify and clean up
m.UnsetStubs()
m.VerifyAll()
m.ResetAll()
Normally, both tests pass, but if I sneak in a typo on my second tests, I get this output when running the tests:
$ ./check_es_docs.test.py
FE
======================================================================
ERROR: test_main_returnsUnknown_ifCalculator_returnsMinus1 (__main__.Main)
If it can't get the current value from ES, print an error message and exit 3
----------------------------------------------------------------------
Traceback (most recent call last):
File "./check_es_docs.test.py", line 13, in test_main_returnsUnknown_ifCalculator_returnsMinus1
m.StubOutWithMock(check_es_insert,"getArgs")
File "/usr/local/lib/python2.7/dist-packages/mox-0.5.3-py2.7.egg/mox.py", line 312, in StubOutWithMock
raise TypeError('Cannot mock a MockAnything! Did you remember to '
TypeError: Cannot mock a MockAnything! Did you remember to call UnsetStubs in your previous test?
======================================================================
FAIL: test_main_doesWhatPrintAndExitSays_inNormalConditions (__main__.Main)
If getCurrent returns a positive value, main() should print the text and exit with the code Calculator.printandexit() says
----------------------------------------------------------------------
Traceback (most recent call last):
File "./check_es_docs.test.py", line 69, in test_main_doesWhatPrintAndExitSays_inNormalConditions
check_es_docs.main()
File "/home/radu/check_es_docs.py", line 25, in main
check_es_insert.printer("Total number of documents in Elasticsearch is %d | 'es_docs'=%d;%d;%d;;" % (result,result,cmdline['warning'],cmdline['critical']))
File "/usr/local/lib/python2.7/dist-packages/mox-0.5.3-py2.7.egg/mox.py", line 765, in __call__
return mock_method(*params, **named_params)
File "/usr/local/lib/python2.7/dist-packages/mox-0.5.3-py2.7.egg/mox.py", line 1002, in __call__
expected_method = self._VerifyMethodCall()
File "/usr/local/lib/python2.7/dist-packages/mox-0.5.3-py2.7.egg/mox.py", line 1060, in _VerifyMethodCall
raise UnexpectedMethodCallError(self, expected)
UnexpectedMethodCallError: Unexpected method call. unexpected:- expected:+
- printer.__call__("Total number of documents in Elasticsearch is 3 | 'es_docs'=3;5;7;;") -> None
? -
+ printer.__call__("Total nuber of documents in Elasticsearch is 3 | 'es_docs'=3;5;7;;") -> None
----------------------------------------------------------------------
Ran 2 tests in 0.002s
FAILED (failures=1, errors=1)
The first test should have passed with no error, since it wasn't changed a bit. check_es_insert.getArgs() shouldn't be a MockAnything instance, and I didn't forget to call UnsetStubs. I've searched quite a lot and I didn't find other people with the same problem. So I guess I'm missing something pretty obvious...
Additional info:
check_es_docs is the script I'm testing
check_es_insert is another script from which I'm importing a lot of stuff
I've tried putting UnsetStubs() after VerifyAll() with the same results
I've tried initializing the mox.Mox() object from the SetUp method, and also putting the cleanup stuff in TearDown, with the same results

Answer 1

I would recommend putting all of your tests into test classes that extend TestCase and then add in an UnsetStubs in the tearDown method:
from unittest import TestCase
import mox
class MyTestCasee(TestCase):
def __init__(self, testCaseName):
self.m = mox.Mox()
TestCase.__init__(self, testCaseName)
def tearDown(self):
self.m.UnsetStubs()
def test_main_returnsUnknown_ifCalculator_returnsMinus1(self):
self.m.StubOutWithMock(check_es_insert,"getArgs")
check_es_insert.getArgs(\
'Nagios plugin for checking the total number of documents stored in Elasticsearch')\
.AndReturn({ 'critical' : 7, 'warning' : 5, 'address' : 'myhost:1234', 'file' : '/tmp/bla'})
################
#some other mocking here, not relevant, I think
################
self.m.ReplayAll()
#now let's test
check_es_docs.main()
#verify and cleanup
self.m.VerifyAll()
def test_main_doesWhatPrintAndExitSays_inNormalConditions(self):
self.m.StubOutWithMock(check_es_insert,"getArgs")
check_es_insert.getArgs(\
'Nagios plugin for checking the total number of documents stored in Elasticsearch')\
.AndReturn({ 'critical' : 7, 'warning' : 5, 'address' : 'myhost:1234', 'file' : '/tmp/bla'})
################
#some other mocking here, not relevant, I think
################
self.m.ReplayAll()
#now let's test
check_es_docs.main()
#verify and clean up
self.m.VerifyAll()
self.m.ResetAll()

Answer 2

You can also use mox.MoxTestBase, which sets up self.mox and calls VerifyAll() on tearDown.
class ClassTestTest(mox.MoxTestBase):
def test():
m = self.mox.CreateMockAnything()
m.something()
self.mox.ReplayAll()
m.something() # If this line is removed the test will fail