Unit testing twisted.web.client.Agent's without the network - unit-testing

Question

I've not done any twisted now for a couple of years and have started using the newer Agent style of client http calls. Using Agent has been OK, but testing is confusing me (it's twisted after all).
I've been through the https://twistedmatrix.com/documents/current/core/howto/trial.html docs and the APIs on trial tools and Agent itself. Also numerous searches.
I've gone with faking out Agent, as I don't need to test that. But then because of the steps to handle the processing and response of an Agent request, my test code has got nasty, implementing the nested layers of the Agent, protocol, etc. Where should I draw the line here and are there some utils I haven't found?
Here's a minimal example (naive implementation of SUT):
from twisted.web.client import Agent, readBody
from twisted.internet import reactor
import json
class SystemUnderTest(object):
def __init__(self, url):
self.url = url
def action(self):
d = self._makeAgent().request("GET", self.url)
d.addCallback(self._cbSuccess)
return d
def _makeAgent(self):
''' It's own method so can be overridden in tests '''
return Agent(reactor)
def _cbSuccess(self, response):
d = readBody(response)
d.addCallback(self._cbParse)
return d
def _cbParse(self, data):
self.result = json.loads(data)
print self.result
with the test module:
from twisted.trial import unittest
from sut import SystemUnderTest
from twisted.internet import defer
from twisted.test import proto_helpers
class Test(unittest.TestCase):
def test1(self):
s_u_t = ExtendedSystemUnderTest(None)
d = s_u_t.action()
d.addCallback(self._checks, s_u_t)
return d
def _checks(self, result, s_u_t):
print result
self.assertEqual({'one':1}, s_u_t.result)
class ExtendedSystemUnderTest(SystemUnderTest):
def _makeAgent(self):
return FakeSuccessfulAgent("{'one':1}")
## Getting ridiculous below here...
class FakeReason(object):
def check(self, _):
return False
def __str__(self):
return "It's my reason"
class FakeResponse(object):
''' Implementation of IResponse '''
def __init__(self, content):
self.content = content
self.prot = proto_helpers.StringTransport()
self.code = 200
self.phrase = ''
def deliverBody(self, prot):
prot.makeConnection(self.prot)
prot.dataReceived(self.content)
# reason = FakeReason()
# prot.connectionLost(reason)
class FakeSuccessfulAgent(object):
''' Implementation of IAgent '''
def __init__(self, response):
self.response = response
def request(self, method, url):
return defer.succeed(FakeResponse(self.response))

Answer 1

but testing is confusing me (it's twisted after all).
Hilarious.
class ExtendedSystemUnderTest(SystemUnderTest):
def _makeAgent(self):
return FakeSuccessfulAgent("{'one':1}")
I suggest you make the agent to use a normal parameter. This is more convenient than a private method like _makeAgent. Composition is great. Inheritance is meh.
class FakeReason(object):
...
There's no reason to make a fake of this. Just use twisted.python.failure.Failure. You don't have to fake every object in the test. Just the ones that get in your way if you don't fake them.
class FakeResponse(object):
...
This is likely good and necessary.
class FakeSuccessfulAgent(object):
...
This is most likely necessary as well. You should make it actually be more like an IAgent implementation though - declare that it implements the interface, use zope.interface.verify.verify{Class,Object} to make sure you get the implementation write, etc (eg request has the wrong signature now).
There's actually a ticket for adding all of these testing tools to Twisted itself - https://twistedmatrix.com/trac/ticket/4024. So I don't think you're actually confused, you're basically on the same track as the project itself. You're just suffering from the fact that Twisted hasn't already done all of this work for you.
Also, note that instead of:
class Test(unittest.TestCase):
def test1(self):
s_u_t = ExtendedSystemUnderTest(None)
d = s_u_t.action()
d.addCallback(self._checks, s_u_t)
return d
You can write something like this instead (and it is preferable):
class Test(unittest.TestCase):
def test1(self):
s_u_t = ExtendedSystemUnderTest(None)
d = s_u_t.action()
self._checks(s_u_t, self.successResultOf(d))
This is because your fake implementation of IAgent is synchronous. You know it is synchronous. By the time request returns, the Deferred it is returning has a result already. Writing the test this way means you can simplify your code a bit (ie, you can ignore the asynchronousness of it to some extent - because it isn't) and it avoids running the global reactor which is what returning a Deferred from a test method in trial does.