I am a true lover of performance, but I am aware about the importance of execute automated tests too (until better formal verification methods become mainstream).
The drawback is that when you try to design for testability you start to introduce interfaces to represent your dependencies. As you know, interfaces make your invocations dynamic dispatched over that dependency and reduce the optimization opportunities at compile-time.
class MyDependency
{
void someMethod();
}
class MyUnit
{
// Concrete type reference of MyDependency allows
// to bypass the dynamic dispatch
this(MyDependency mayBeAMock)
{
mayBeAMock.someMethod();
}
unittest
{
// Now how can I get a mock of my dependency without to instantiate it.
auto dep = someBlackMagic.getMock();
auto uut = new MyUnit(dep);
}
}
Does exist a better way to unit test classes without take the dynamic dispatch cost at production. If need, I would can to afford that cost for the unit testing execution but not for the production release.
Im am interested in D and C++ solutions.
One technique I use in D very often is to do compile-time policy replacement:
private struct MyUtilImpl ( HTTPClient )
{
void foo ( )
{
HTTPClient.makeRequest("url");
}
}
version (unittest)
alias MyUtil = MyUtilImpl!FakeHTTPClient;
else
alias MyUtil = MyUtilImpl!RealHTTPClient;
It is very similar to classical dependency injection in spirit but instead of mocking I/O utilities and swapping via interfaces, I do the same at compile time using template parameters.
It also has the benefit of not needing to change anything in rest of the program to start passing interface around.
2 ideas.
You could either make MyUnit a template and pass in a class that implements (or mocks) "someMethod"
Or, if your unit tests rebuild the class you are testing, you could define the Mock and the real version of the class in separate files and conditionally include / link to the one you want.
Of course you have to be careful not to make your tested version too different from the non-test version.
Related
Suppose I have code like below. http_client is an external dependency (a 3rd party API) I don't have control over. transaction_handler is a class I control and would like to write unit tests for.
// 3rd party
class http_client
{
public:
std::string get(std::string url)
{
// makes an HTTP request and returns response content as string
// throws if status code is not 200
}
};
//code to be tested
enum class transaction_kind { sell, buy };
enum class status { ok, error };
class transaction_handler
{
private:
http_client client;
public:
status issue_transaction(transaction_kind action)
{
try
{
auto response =
client.get(std::string("http://fake.uri/") +
(action == transaction_kind::buy ? "buy" : "sell"));
return response == "OK" ? status::ok : status::error;
}
catch (const std::exception &)
{
return status::error;
}
}
};
Because http_client makes network calls I would like to be able to substitute it in my tests with a mock implementation which cuts off the network and allows for testing different conditions (ok/error/exception). Because transaction_handler is supposed to be internal I can modify it to make it testable but I wouldn't want to go over the border (i.e. I would like to avoid pointers or dynamic polymorphism if possible). Ideally I would like to use a kind of dependency injection where my tests would inject a mock http_client. I don't think I can/want to use a 'poor man's DI' where I would create an http_client implementation in the caller and pass it to the transaction_handler (by const reference? std::shared_ptr?) - because I don't control the http_client I would have to come up with an interface and -in the product code - I would have to wrap the http_client in a wrapper class that implements this interface and forwards the calls to the actual/wrapped http_client instance. In the test code I would create a mock implementation of that interface. The interface would have to be a pure abstract method which entails using runtime polymorphism which I wanted to avoid. Another option is to use templates. If I changed the transaction_handler class to look as follows:
template <typename T = http_client>
class transaction_handler
{
private:
T client;
public:
transaction_handler(const std::function<T()> &create) : client(create())
{}
status issue_transaction(transaction_kind action)
{
// same as above, omitted for brevity
}
}
I could now create a mock http_client class:
class http_client_mock
{
public:
std::string get(std::string url)
{
return std::string("OK");
}
};
and create the transaction_class object in my tests like this:
transaction_handler<http_client_mock> t(
[]() -> http_client_mock { return http_client_mock(); });
while I could use the following in my product code:
transaction_handler<> t1(
[]() -> http_client { return http_client(); });
While it seems to work and fullfill most of my requirements (even though I don't like the fact that the code instantiating transaction_handler need to be aware of the http_client type - maybe it can be somehow hidden as a factory class) - does it make sense at all? Or may be there are better ways of doing this kind of things? I spent a considerable amount of time looking for some simple DI patterns to make unit testing easier bud had hard time finding something that would suit my needs. Also, my background is mostly C so maybe I approach the problem from a wrong angle?
I'm maintaining a DI library, and your case is really interesting for me.
Mocking is about dynamic polymorphism or compile time mocking (at least when using C++). You pay 1 indirection to obtain ability to inject what you want (dependence only on 1 interface).
If you want to make code testable, the best way is using interfaces (pure virtual classes in C++ wich does not have interfaces) and inject dependencies only through constructor.
If you really want to avoid polymorphism (or you can't because of external API) you could still accept to have some code that is not fully testable.
Conventional way of doing things:
class ConcreteHttpClient : public virtual AbstractHttpClient { /*...*/}
class MockHttpClient : public virtual AbstractHttpClient{ /*...*/ }
You just choose what to inject based on needs ( I intentionally use "new" instead of showing some DI framwork at work).
Production code.
new TransactionHandler ( static_cast< AbstractService>( ConcreteService));
Unit testing the transaction handler
new TransactionHandler ( static_cast< AbstractService>( MockService));
If you later need to test some class using the transaction handler and the transaction handler implements a interface
class TransactionHandler: public virtual AbstractTransactionHandler { /*...*/}
You have just to create a TransactionHandlerMock inheriting from AbstractTransactionHandler
When you use interfaces the advantage is that you can use a Dependency Injection framework to avoid poor man's injection.
Compile time mocking.
What you proposed is a viable alternative, basically you assume a "static polymorphism" thanks to templates
Production code:
template <typename T = http_client>
class transaction_handler{/*...*/};
new transaction_handler( /*...*/ );
Unit test code:
using transaction_handler_mocked = transaction_handler< http_client_mock>;
new transaction_handler_mocked( /*...*/ );
However this has few Issues:
You are depending on "transaction_handler" type on each part of your production code so if you change it you have to recompile every file depending on it.
You can't inject a mocked handler as a mock itself, unless you change all classes depending on it to become templates accepting the handler.
Point number 2 means that basically in a complex project you have each class depending on each other, increasin compile times and forcing to whole recompiles of your project just for little changes
You are not using a Interface (pure virtual class) that means that you can still accidentally access fields or members of template parameters that were not intended to be accessed making your debuggin harder.
Other alternatives
Provide mock at link time, you don't have to mock a whole 3rd party library. Just the classes you are testing. Most IDEs are not thinked to work that way, but probably you can work around with some bash script or custom makefile. (In example, I do that for testing code dependent on C functions, in particular OpenGL)
Wrap interesting functionalities of libraries you want to test behind a class implementing a pure virtual class (don't know why you want to avoid it). You have good chances that you don't need to wrap all methods and you'll end with a smaller API to test against (just wrap parts you need to use, don't start up front wrapping the whole library)
Use a mocking framework, and possibly a dependency Injection framework.
Just write test routines that match whichever exports of http_client you're using. You're source will be linked in preference to any lib.
I think it's a good idea to mock it. Since http_client is an external dependency, I chose Typemock Isolator++ to handle with it. Look at the code below:
TEST_METHOD(FakeHttpClient)
{
//Arrange
string okStr = "OK";
http_client* mock_client = FAKE_ALL<http_client>();
WHEN_CALLED(mock_client->get(ANY_VAL(std::string))).Return(&okStr);
//Act
transaction_handler my_handler;
status result = my_handler.issue_transaction(transaction_kind::buy);
//Assert
Assert::AreEqual((int)status::ok, (int)result);
}
Method FAKE_ALL<> allows me to set the behavior of all http_client instances, so no injection needed. Simple API, the code looks accurate, and you don't need to change the production code.
Hope it helps!
I'm attempting to help design some unit tests around controllers in a Qt C++ application.
To be frank, I have two large drawbacks. One, my testing background is heavily based on .NET projects, so my knowledge of best practice in the c++ world is slim at best. Two, the designer of the application I am looking at did not architect the code with unit testing in mind.
One specific point point, I'm looking at a controller class that includes boost/filesystem/operations.hpp. The controller constructor goes on to check directory existence and create directories using functions from the boost filesystem code.
Is there any way to overload or mock this behavior? I'm used to setting up an IoC container or at least dependency injected constructors in .NET, and then being able to pass mock objects in the unit test code. I'm not sure how a templated header file would work with that concept, though, or if it is even typical practice in c++.
Currently, I have no flexibility to suggest code changes, as there is a release build coming up this week. But after that, if there are some simple code changes that could improve testability, that is definitely an option. Ideally, there would be a way to overload the filesystem functions in the unit test framework as is, though.
We ended up creating a generic file system wrapper that calls Boost filesystem and accepting it as a parameter to our class constructors so we could send in mock versions at unit test time.
I understand the thought to not mock this, but I think there is value in fast unit tests for our CI environment to run at check in time as well as tests that actually hit the file system.
If you think about it, the only reason why you need to inject an instance of, suppose, IFilesystem into your classes, is to mock it in tests. No part of your non-test codebase gonna use anything but the real filesystem anyway, so it is safe to assume that you can inject not an object but a type into your classes and use them freely within your codebase without type clashes.
So, suppose you have class of interest
struct BuildTree
{
BuildTree(std::string_view dirname) { /*...*/ }
bool has_changed()
{
// iterates through files in directory
// recurs into directories
// checks the modification date against lastModified_
// uses boost::filesystem::last_write_time(), etc.
}
private:
boost::filesystem::file_time_type lastModified_;
};
Now, you could inject a type into the class. This type will be a class with a bunch of static methods. There will be a RealFilesystem type that will redirect to the boost::filesystem methods, and there will be a SpyFilesystem.
template <class Filesystem>
struct BuildTree
{
// ...
bool has_changed()
{
// uses Filesystem::last_write_time(), etc.
}
private:
typename Filesystem::file_time_type lastModified_;
};
The SpyFilesystem will resemble a PIMPL idiom, in that the static methods will redirect calls to the actual implementation.
struct SpyFilesystemImpl;
struct SpyFilesystem
{
using file_time_type = typename SpyFilesystemImpl::file_time_type;
static file_time_type last_time_write(std::string_view filename)
{
return instance.last_time_write(filename);
}
// ... more methods
static SpyFilesystemImpl instance;
};
SpyFilesystemImpl SpyFilesystem::instance{};
// No warranty of completeness provided
struct SpyFilesystemImpl
{
using file_time_type = std::chrono::system_clock::time_point;
void create_directory(std::string_view path) { /*...*/ }
void touch(std::string_view filename)
{
++lastModified_[filename];
}
file_time_type last_time_write(std::string_view filename)
{
return std::chrono::system_clock::time_point{lastModified_[filename]};
}
private:
std::unordered_map<std::string, std::chrono::seconds> lastModified_;
};
Finally, inside each of your tests you would prepare an instance of SpyFilesystemImpl, assign it to the SpyFilesystem::instance and then instantiate your class with SpyFilesystem. like so
// ... our SpyFilesystem and friends ...
// Google Test framework
TEST(BuildTree, PickUpChanges)
{
SpyFilesystemImpl fs{};
fs.create_directory("foo");
fs.touch("foo/bar.txt");
SpyFilesystem::instance = fs;
BuildTree<SpyFilesystem> tree("foo");
EXPECT_FALSE(tree.has_changed());
SpyFilesystem::instance.touch("foo/bar.txt");
EXPECT_TRUE(tree.has_changed());
}
This approach has an advantage that there will be no run-time overhead in the resulting binary (provided that optimizations are enabled). However, it requires more boilerplate code, which might be a problem.
It seems reasonable to me to consider boost::filesystem as an
extension of the standard libraries. So you mock it (or not) in
exactly the same way you mock something like std::istream.
(Generally, of course, you don't mock it, but rather your test
framework provides the necessary environment: the files you need
to read with std::istream, the directories, etc. for
boost::filesystem.)
So I'm wondering about how unit testing works in regards to dealing external dependencies. Here and elsewhere I've become familiar with dependency injection, and how that allows us to test a unit (A) of code. However, I'm confused about how to test other units (B and C) which are now possess the external dependency so they can inject it into the original unit (A).
For example, say some class Foo uses an external dependency...
class Foo
{
private ExternalDependency ed;
public int doSomethingWithExternalDependency() {...}
}
And class Bar makes use off Foo...
class Bar
{
public int doSomethingWithFoo
{
Foo f = new Foo();
int x = f.doSomethingWithExternalDependency();
// Do some more stuff ...
return result;
}
}
Now, I know that I can use dependency injection so that I can test Foo, but then how do I test Bar? I guess, I can, again, use dependency injection, but at some point some unit needs to actually create the external dependency; so how do I test that unit?
The examples you provide do not use Dependency Injection. Instead, Bar should use Constructor Injection to get a Foo instance, but there's no point in injecting a concrete class. Instead, you should extract an interface from Foo (let's call it IFoo) and inject that into Bar:
public class Bar
{
private IFoo f;
public Bar(IFoo f)
{
this.f = f;
}
public int doSomethingWithFoo
{
int x = this.f.doSomethingWithExternalDependency();
// Do some more stuff ...
return result;
}
}
This enables you to always decouple consumers and dependencies.
Yes, there will still be a place where you must compose the entire application's object graph. We call this place the Composition Root. It's a application infrastructure component, so you don't need to unit test it.
In most cases you should consider using a DI Container for that part, and then apply the Register Resolve Release pattern.
Keep in mind the difference between unit testing and integration testing. In the former, the dependency would be mocked whereby it provides expected behavior for the purpose of testing the class which consumes the dependency. In the latter, an actual instance of the dependency is initialized to see if the whole thing works end-to-end.
In order to use Dependency Injection, your classes would have their dependencies injected into them (several ways to do that - constructor injection, property injection) and would not instantiate them themselves, as you do in your examples.
Additionally, one would extract the interface of each dependency to help with testability and use the interface instead of an implementation type as the dependency.
class Foo
{
private IExternalDependency ed;
public int doSomethingWithExternalDependency() {...}
public Foo(IExternalDependency extdep)
{
ed = extdep;
}
}
What most people do is use a mocking framework to mock the dependencies when testing.
You can mock any object that the class under test depends on (including behavior and return values) - pass the mocks to the class as its dependencies.
This allows you to test the class without relying on the behavior of its (implemented) dependencies.
In some cases, you may want to use fakes or stubs instead of a mocking framework. See this article by Martin Fowler about the differences.
As for getting all the dependencies, all the way down - one uses an IoC container. This is a registry of all of the dependencies in your system and understands how to instantiate each and every class with its dependencies.
When you unit test a class, you should mock its dependencies, to test your class in isolation -- this is regardless of dependency injection.
The answer to your question about Bar is: yes, you should inject Foo. Once you go down the DI path, you will use it across your entire stack. If you really need a new Foo for every doSomethingWithFoo call, you might want to inject a FooFactory (which you can then mock for testing purposes), if you want a single Bar to use many Foos.
I'd like to stress out that in case of unit testing you should have two separate sets of tests: one for Foo.doSomethingWithExternalDependency and another one for Bar.doSomethingWithFoo. In the latter set create mock implementaion of Foo and you test just doSomethingWithFoo assuming that doSomethingWithExternalDependency works properly. You test doSomethingWithExternalDependency in a separate test set.
It known that in C++ mocking/faking nonvirtual methods for testing is hard. For example, cookbook of googlemock has two suggestion - both mean to modify original source code in some way (templating and rewriting as interface).
It appear this is very bad problem for C++ code. How can be done best if you can't modify original code that needs to be faked/mocked? Duplicating whole code/class (with it whole base class hierarchy??)
One way that we sometimes use is to split the original .cpp file into at least two parts.
Then the test apparatus can supply its own implementations; effectively using the linker to do the dirty work for us.
This is called the "Link Seam" in some circles.
I followed the Link Seam link from sdg's answer. There I read about different types of seams, but I was most impressed by Preprocessing Seams. This made me think about exploiting further the preprocessor. It turned out that it is possible to mock any external dependency without actually changing the calling code.
To do this, you have to compile the calling source file with a substitute dependency definition.
Here is an example how to do it.
dependency.h
#ifndef DEPENDENCY_H
#define DEPENDENCY_H
class Dependency
{
public:
//...
int foo();
//...
};
#endif // DEPENDENCY_H
caller.cpp
#include "dependency.h"
int bar(Dependency& dependency)
{
return dependency.foo() * 2;
}
test.cpp
#include <assert.h>
// block original definition
#define DEPENDENCY_H
// substitute definition
class Dependency
{
public:
int foo() { return 21; }
};
// include code under test
#include "caller.cpp"
// the test
void test_bar()
{
Dependency mockDependency;
int r = bar(mockDependency);
assert(r == 42);
}
Notice that the mock does not need to implement complete Dependency, just the minimum (used by caller.cpp) so the test can compile and execute.
This way you can mock non-virtual, static, global functions or almost any dependency without changing the productive code.
Another reason I like this approach is that everything related to the test is in one place. You don't have to tweak compiler and linker configurations here and there.
I have applied this technique successfully on a real world project with big fat dependencies.
I have described it in more detail in Include mock.
Code has to be written to be testable, by whatever test techniques you use. If you want to test using mocks, that means some form of dependency injection.
Non-virtual calls with no dependence on a template parameter pose the same problem as final and static methods in Java[*] - the code under test has explicitly said, "I want to call this code, not some unknown bit of code that's dependent in some way on an argument". You, the tester, want it to call different code under test from what it normally calls. If you can't change the code under test then you, the tester, will lose that argument. You might as well ask how to introduce a test version of line 4 of a 10-line function without changing the code under test.
If the class to be mocked is in a different TU from the class under test, you can write a mock with the same name as the original and link that instead. Whether you can generate that mock using your mocking framework in the normal way, I'm not so sure.
If you like, I suppose it's a "very bad problem for C++" that it's possible to write code that's hard to test. It shares this "problem" with a great number of other languages...
[*] My Java knowledge is quite low-power. There may be some clever way of mocking such methods in Java, which aren't applicable to C++. If so, please disregard them in order to see the analogy ;-)
I think it is not possible to do it with standard C++ right now (but lets hope that a powerful compile-time reflection will come to C++ soon...). However, there are a number of options for doing so.
You might have a look at Injector++. It is Windows only right now, but plans to add support for Linux & Mac.
Another option is CppFreeMock, which seems to work with GCC, but has no recent activities.
HippoMocks also provide such ability, but only for free functions. It doesn't support it for class member functions.
I'm not completely sure, but it seems that all the above achieve this with overwriting the target function at runtime so that it jumps to the faked function.
The there is C-Mock, which is an extension to Google Mock allowing you to mock non-virtual functions by redefining them, and relying on the fact that original functions are in dynamic libraries. It is limited to GNU/Linux platform.
Finally, you might also try PowerFake (for which, I'm the author) as introduced here.
It is not a mocking framework (currently) and it provides the possibility for replacing production functions with test ones. I hope to be able to integrate it to one or more mocking frameworks; if not, it'll become one.
Update: It has an integration with FakeIt.
Update 2: Added support for Google Mock
It also overrides the original function during linking (so, it won't work if a function is called in the same translation unit in which it is defined), but uses a different trick than C-Mock as it uses GNU ld's --wrap option. It also needs some changes to your build system for tests, but doesn't affect the main code in any way (except if you are forced to put a function in a separate .cpp file); but support for easily integrating it into CMake projects is provided.
But, it is currently limited to GCC/GNU ld (works also with MinGW).
Update: It supports GCC & Clang compilers, and GNU ld & LLVM lld linkers (or any compatible linker).
#zaharpopov you can use Typemock IsolatorPP to create mocks of non-virtual class and methods without changing your code (or legacy code).
for example if you have a non-virtual class called MyClass:
class MyClass
{
public:
int GetResult() { return -1; }
}
you can mock it with typemock like so:
MyClass* fakeMyClass = FAKE<MyClass>();
WHEN_CALLED(fakeMyClass->GetResult()).Return(10);
By the way the classes or methods that you want to test can also be private as typemock can mock them too, for example:
class MyClass
{
private:
int PrivateMethod() { return -1; }
}
MyClass* myClass = new MyClass();
PRIVATE_WHEN_CALLED(myClass, PrivateMethod).Return(1);
for more information go here.
You very specifically say "if you can't modify original code", which the techniques you mention in your question (and all the other current "answers") do.
Without changing that source, you can still generally (for common OSes/tools) preload an object that defines its own version of the function(s) you wish to intercept. They can even call the original functions afterwards. I provided an example of doing this in (my) question Good Linux TCP/IP monitoring tools that don't need root access?.
That is easier then you think. Just pass the constructed object to the constructor of the class you are testing. In the class store the reference to that object. Then it is easy to use mock classes.
EDIT :
The object that you are passing to the constructor needs an interface, and that class store just the reference to the interface.
struct Abase
{
virtual ~Abase(){}
virtual void foo() = 0;
};
struct Aimp : public Abase
{
virtual ~Aimp(){}
virtual void foo(){/*do stuff*/}
};
struct B
{
B( Aimp &objA ) : obja( objA )
{
}
void boo()
{
objA.foo();
}
Aimp &obja;
};
int main()
{
//...
Aimp realObjA;
B objB( realObjA );
// ...
}
In the test, you can pass the mock object easy.
I used to create an interface for the parts I needed to mock. Then I simply created a stub class that derived from this interface and passed this instance to my classes under test. Yes, it is a lot of hard work, but I found it worth it for some circumstances.
Oh, by interface I mean a struct with only pure virtual methods. Nothing else!
To be able to unit test my C++ code I usually pass the constructor of the class under test one or several objects that can be either "production code" or fake/mock objects (let's call these injection objects). I have done this either by
Creating an interface that both the "production code" class and the fake/mock class inherits.
Making the class under test a template class that takes the types of the injection objects as template parameters, and instances of the injection objects as parameters to the constructor.
Some random thoughts:
Until we have concepts (C++0x), only documentation and parameter naming will hint what to provide the class under test (when using templates).
It is not always possible to create interfaces for legacy code
The interface is basically only created to be able to do dependency injection
In the same way: templating the class under test is done only to enable dependency injection
What are your thoughts? Are there other solutions to this problem?
With C++, there's another option - you give your mock classes exact same names as the real classes, and when linking your unit tests, just link them with mock object/library files instead of real ones.
I think interface option is better, but one doesn't have to create common base class just for test. You can inherit your mock class from production class and override necessary methods. You'll have to make the methods virtual though, but that's how tools like mockpp work and they also allow automate this process a little bit.
Templates will have slightly less performance penalties for runtime (less indirections, less calls, more inline optimizations), but will make you suffer a very high penalty for compilation times...
I think that for this purpose, interfaces are better (until we have concepts in C++0x TR1)... unless if you can't slow down some "bottleneck code". Interfaces are more dynamic and switchable at run-time.
Remember that you can construct your class with default injection objects (the real ones), but you can have factories that inject the mock ones on your tests... you don't even need to subclass.
Don't know if it helps, but you can have template constructors:
struct Class_Under_Test
{
template <typename Injected>
Class_Under_Test()
{
...
// and even specialize them
template <>
Class_Under_Test <A_Specific_Injection_Class>
{
...
Only the one that is actually used will get included.