I have some written a number of unit tests that test a wrapper around a FTP server API.
Both the unit tests and the FTP server are on the same machine.
The wrapper API gets deployed to our platform and are used in both remoting and web service scenarios. The wrapper API essentially takes XML messages to perform tasks such as adding/deleting/updating users, changing passwords, modifying permissions...that kinda thing.
In a unit test, say to add a user to a virtual domain, I create the XML message to send to the API. The API does it's work and returns a response with status information about whether the operation was successful or failed (error codes, validation failures etc).
To verify whether the API wrapper code really did do the right thing (if the response indicated success), I invoke the FTP server's COM API and query its store directly to see if, for example when creating a user account, the user account really did get created.
Does this smell bad?
Update 1: #Jeremy/Nick: The wrapper is the focus of the testing, the FTP server and its COM API are 3rd party products, presumably well tested and stable. The wrapper API has to parse the XML message and then invoke the FTP server's API. How would I verify, and this may be a silly case, that a particular property of the user account is set correctly by the wrapper. For example setting the wrong property or attribute of an FTP account due to a typo in the wrapper code. A good example being setting the upload and download speed limits, these may get transposed in the wrapper code.
Update 2: thanks all for the answers. To the folks who suggested using mocks, it had crossed my mind, but the light hasn't switched on there yet and I'm still struggling to get my head round how I would get my wrapper to work with a mock of the FTP server. Where would the mocks reside and do I pass an instance of said mocks to the wrapper API to use instead of calling the COM API? I'm aware of mocking but struggling to get my head round it, mostly because I find most of the examples and tutorials are so abstract and (I'm ashamed to say) verging on the incomprehensible.
You seem to be mixing unit & component testing concerns.
If you're unit-testing your wrapper, you should use a mock FTP server and don't involve the actual server. The plus side is, you can usually achieve 100% automation like this.
If you're component-testing the whole thing (the wrapper + FTP server working together), try to verify your results at the same level as your tests i.e. by means of your wrapper API. For example, if you issue a command to upload a file, next, issue a command to delete/download that file to make sure that the file was uploaded correctly. For more complex operations where it's not trivial to test the outcome, then consider resorting to the COM API "backdoor" you mentioned or perhaps involve some manual verification (do all of your tests need to be automated?).
To verify whether the API wrapper code really did do the right thing (if the response indicated success), I invoke the FTP server's COM API
Stop right there. You should be mocking the FTP server and the wrapper should operate against the mock.
If your test runs both the wrapper and the FTP server, you are not Unit Testing.
To test your wrapper with a mock object, you can do the following:
Write a COM object that has the same interface as the FTP server's COM API. This will be your mock object. You should be able to interchange the real FTP server and your mock object by passing the interface pointer of either to your wrapper by means of dependency injection.
Your mock object should implement hard-coded behaviour based on the methods called on its interface (which mimics FTP server API) and also based on the argument values used:
For example, if you have an UploadFile method you can blindly return a success result and perhaps store the file name that was passed in in an array of strings.
You could simulate an upload error when you encounter a file name with "error" in it.
You could simulate latency/timeout when you encounter a file name with "slow" in it.
Later on, the DownloadFile method could check the internal string array to see if a file with that name was already "uploaded".
The pseudo-code for some test cases would be:
//RealServer theRealServer;
//FtpServerIntf ftpServerIntf = theRealServer.getInterface();
// Let's test with our mock instead
MockServer myMockServer;
FtpServerIntf ftpServerIntf = myMockServer.getInterface();
FtpWrapper myWrapper(ftpServerIntf);
FtpResponse resp = myWrapper.uploadFile("Testing123");
assertEquals(FtpResponse::OK, resp);
resp = myWrapper.downloadFile("Testing123");
assertEquals(FtpResponse::OK, resp);
resp = myWrapper.downloadFile("Testing456");
assertEquals(FtpResponse::NOT_FOUND, resp);
resp = myWrapper.downloadFile("SimulateError");
assertEquals(FtpResponse::ERROR, resp);
I hope this helps...
I agree with Nick and Jeremy about not touching the API. I would look at mocking the API.
http://en.wikipedia.org/wiki/Mock_object
If it's .NET you can use:
Moq: http://code.google.com/p/moq/
And a bunch of other mocking libraries.
What are you testing the wrapper or the API. The API should work as is, so you don't need to test it I would think. Focus your testing efforts on the wrapper and pretend like the API doesn't exist, when I write a class that does file access I don't unit test the build in streamreader...I focus on my code.
I would say your API should be treated just like a database or a network connection when testing. Don't test it, it isn't under your control.
It doesn't sound like you're asking "Should I test the API?" — you're asking "Should I use the API to verify whether my wrapper is doing the right thing?"
I say yes. Your unit tests should assert that your wrapper passes along the information reported by the API. In the example you give, for instance, I don't know how you would avoid touching the API. So I don't think it smells bad.
The only time I can think of when it might make sense to dip into the lower level API to verify results if if the higher-level API is write-only. For example, if you can create a user using the high-level API, then there should be a high-level API to get the user accounts, too. Use that.
Other folks have suggested mocking the lower-level API. That's good, if you can do it. If the lower-level component is mocked, checking the mocks to make sure the right state is set should be okay.
Related
It's tremendously helpful that there's a tool to generate mock versions of the client stubs. Testing the server side is causing me tons of headache at the moment. Enough headache where I feel like I must be doing something fundamentally wrong.
I may be misreading the following, but the end2end tests, including 'mock_test' seem to be using an actual client-server connection to drive testing. They may mock out the client, or mock out the client readers/writers to see the response from the server, but it's not clear to me how to test the server in isolation.
What I want to be able to do: I have some Service implentation that inherits from the gRPC generated class "Service." suppose that service exposes an interface ::grpc::Status Foo(::grpc::ServerContext* context, const CommandMessage* request, ::grpc::ServerWriter<CommandResponse>* writer); My gut for writing unit tests is saying to pass in a mock "ServerWriter" class and expect 'Write' is called when appropriate. But ServerWriter is marked final and can't be overridden.
This isn't the first place I've run into trouble with my kind of standard ways of mocking and gRPC's server stuff. The Server class, the ServerBuilder class, etc. I've wrapped so that I could put mock versions of them into tests (to validate that the correct parameters are being passed to my Server when it's being constructed, e.g.)
So I think I'm missing something with grpc then. I just don't know what. Am I supposed to stand up a real server in my unit tests and probe it with a mock client? How do I validate the proper server configurations are being passed, if I have to stand up a test version with test configurations? The code has interface classes and virtual methods, but then the parts that seem exposed for public use don't seem to be easily mockable as I'd expect.
I'm writing a Gradle plugin that interacts with an external HTTP API. This interaction is handled by a single class (let's call it ApiClient). I'm writing some high-level tests that use Gradle TestKit to simulate an entire build that uses the plugin, but I obviously don't want them to actually hit the API. Instead, I'd like to mock ApiClient and check that its methods have been called with the appropriate arguments, but I'm not sure how to actually inject the mocked version into the plugin. The plugin is instantiated somewhere deep within Gradle, and gets applied to the project being executed using its void apply(Project project) method, so there doesn't appear to be a way to inject a MockApiClient object.
Perhaps one way is to manually instantiate a Project, apply() the plugin to it (at which point, I can inject the mocked object because I have control over plugin instantiation), and then programmatically execute a task on the project, but how can I do that? I've read the Gradle API documentation and haven't seen an obvious way.
A worst-case solution will be to pass in a debug flag through the plugin extension configuration, which the plugin will then use to determine whether it should use the real ApiClient or a mock (which would print some easily grep-able messages to the STDOUT). This isn't ideal, though, since it's more fuzzy than checking the arguments actually passed to the ApiClient methods.
Perhaps you could split your plugin into a few different plugins
my-plugin-common - All the common stuff
my-plugin-real-services - Adds the "real" services to the model (eg RealApiClient)
my-plugin-mock-services - Adds "mock" services to the model (eg MockApiClient)
my-plugin - Applies my-plugin-real-services and my-plugin-common
my-plugin-mock - Applies my-plugin-mock-services and my-plugin-common
In the real world, people will only ever apply: 'my-plugin'
For testing you could apply: 'my-plugin-mock'
As title, when to use httptest.Server and httptest.ResponseRecorder?
It seems to me that I can also test my handlers to return correct response using httptest.Server. I can simply start a httptest.Server given with my implementation of handlers, then do validations on the response's body.
Please correct if I'm wrong, I am learning Go + TDD
When you just want to check, if your http.Handler does what it should, you don't need to use httptest.Server. Just call your handler with an httptest.ResponseRecorder instance and check the output as in the example.
The possible uses of httptest.Server are numerous, so here are just a couple that come to my mind:
If your code depends on some external services and APIs, you can use a test server to emulate them. (Although I personally would isolate all code dealing with external data sources and then use them through interfaces, so that I could easily create fake objects for my tests.)
If you work on a client-server application, you can use a test server to emulate the server-side when testing the client-side.
I have some of WebClient Requests in my App and want to check the parsing of the retrieved data in a unit test. How Do I wait in the WP7 Silverlight UnitTestFramework for the event client_DownloadStringCompleted?
My approach to this has been to introduce a wrapper class around WebClient (well, I actually used HttpWebRequest in the end, as WebClient did too much in the UI thread...) implementing an interface. I could then create a FakeWebClient implementing IWebClient, allowing me to validate the URLs that were being fetched, and responding with errors, success cases etc as desired.
It's unfortunate that quite a few APIs in .NET aren't easily testable / fakable :(
Completely wrong approach here. You're unit testing the wrong thing.
What you want to do, is to move your parsing of the data out to another class, and define a interface, say:
interface IWebParser { MyResult Parse(string input); }
and then inject that into your class, and in your DownloadStringCompleted event, call iWebParser.Parse(e.Result).
Now you can test your implementations of the IWebParser. And replace it.
Unit Testing isn't meant to test implementation specific code. You might as well just use a accessor and test a private method then!
Usually you should not test with HTTP requests. But to write such tests anyway eventually this link will help you (asynchronous testing):
http://www.jeff.wilcox.name/2009/03/asynchronous-testing/
I've created web-service in Delphi XE using WSDL importer.
Delphi generated for me module ITransmitter1.pas with
ITransmitter interface and GetITransmitter function.
To use webservice i use:
var Transmitter: ITransmitter;
begin
Transmitter := GetITransmitter(True, '', nil);
Transmitter.Transmit(Memo1.Text, OutXML);
end;
But i cant see anywhere body of method Transmit ...
In ITransmitter.pas i see:
InvRegistry.RegisterInterface(TypeInfo(ITransmitter), 'urn:TransmitterIntf-ITransmitter', 'utf-8');
InvRegistry.RegisterDefaultSOAPAction(TypeInfo(ITransmitter), 'urn:TransmitterIntf-ITransmitter#Transmit');
If i comment this lines i get "interface not supported" error.
As i see here delphi is adding method in RunTime !
How does it work ? Can i add method in runtime to my own class ?
If you created a web service client with the WSDL importer, the generated client code will invoke a method on the server. So in fact, the method 'body' (code) is on the web service server.
Delphi generates a Soap request based on the WSDL, and behind the scenes RTTI (introspection) is used to generate parameters etc. of the web service call as XML. This XML is sent to the server, which executes the method implementation and sends back a Soap response.
Things are opposite if you create the web service server, in this case the Delphi application of course needs to implement all method bodies.
You're in fact calling a method defined in a Interface, which in turn inherits from IInvokable, declared in System.pas.
If you check your source code you'll note that no local object in your project implements the IInvokable interface you're calling, that's because that method is remotely executed in the server.
Before that occurs, there's some pascal code used to create a proper SOAP request to the server, send it and then wait and interpret the server response, consider this implementation details. If you're interested in know a bit more how this works, enable the "use debug .dcus" compiler option, so you can debug inside the VCL/RTL.
Then, as usual, use the StepInto (F7) command to ask the debugger to execute the Transmit method step by step... after some assembler in the TRIO.GenericStub method you'll get to the TRIO.Generic method where the packet is prepared and sent.
For a btSOAP binding I'm using to write this response, the relevant part starts at line 943 in the Rio.pas unit:
try
FWebNode.Execute(Req, Resp);
finally
{ Clear Outbound headers }
FHeadersOutBound.Clear;
end;
THTTPReqResp.Execute then uses wininet.dll functions to perform the connection, send and receive of information with the server using.
There are some levels you can go deep with this... how far you want to go will depend on your interests and the great amount of details are far beyond the scope of my answer here... feel free to post more questions with specific tings you're interested in.
I'm not sure about, but details can change between Delphi versions... I'm using Delphi XE right now.