I am trying to understand what could be the best design that helps in writing testable code. Let's consider below code ..
void Connection::Connect(const string& url)
{
// My code here that I want to test
ThirdpartClass::Connect(modifiedUrl)
// My code here that I want to test
}
In above example ThirdpartClass::Connect(modifiedUrl) is a third part dependency that actually connects to some third-party server which I want to omit as a part of my unit test code and replace it with mock connect call while executing unit tests. My existing code is tightly bound with third party dependency which makes non testable without involving third party. Is there a better design that makes my code test ready.
Low latency is important so run time polymorphism should be a last resort.
I am planning to use google test framework for writing unit test code.
You can give static polymorphism (i.e. templates) a try if you really want to, although I'm not sure the extra complication for the code will be worth it in the end.
First, make your Connection class templated on the type of the class you'll be calling for the connection:
template <typename C = ThirdPartyClass>
class Connection { ... }
Then, your Connect function now looks like this:
void Connection<C>::Connect(const string& url)
{
// My code here that I want to test
C::Connect(modifiedUrl)
// My code here that I want to test
}
Whatever type you pass in as a template parameter to Connection will be used, instead of forcing you to use ThirdPartyClass.
For testing, you can then do the following:
using TestConnection = Connection<MockClass>;
And for actual code, use the ThirdPartyClass instead of the MockClass.
You have to somehow remove the dependency to the third party class when you are testing and unfortunately, you will have to slightly modify your code.
You can create a wrapper around your third party functions. For example:
class ThirdPartyWrapperInterface {
public:
virtual bool Connect(std::string url) = 0;
};
// Use this in production
class ThirdPartyWrapper : public ThirdPartyWrapperInterface {
public:
bool Connect(std::string url) override {
return ThirdpartClass::Connect(modifiedUrl);
}
};
// Use this for test
class MockThirdPartyWrapper : public ThirdPartyWrapperInterface {
public:
MOCK_METHOD(bool, Connect, (std::string), (override));
};
Now your original code should use the wrapper instead and it can select between the two classes using dependency injection as you would normally do when you mock a function.
I understand that runtime polymorphism is not ideal, so you can use templates instead. See here or here for an example.
Related
I am exploring TDD and SOLID principles. Say I have a service that I create a failing test for before writing the implementation.
public interface IService {
bool DoSomething();
}
public class Service : IService {
bool DoSomething() {
...
}
}
[TestMethod]
public void ServiceImplementationTest()
{
var implementation = new Service();
bool result = implementation.DoSomething();
Assert.IsTrue(result);
}
When I first write the test, I'm unaware of the dependencies of this service, so the constructor takes no arguments as I don't need to inject any dependencies.
However, as I write the implementation, I realise I need a certain dependency, and so add a reference to that dependency in the constructor. To keep the test code compiling and failing, I then have to go back to the test, and modify it to create a fake implementation.
public class Service : IService {
public Service(IDependency dependency) {
_dependency = dependency;
}
bool DoSomething() {
... use _dependency ...
return result;
}
}
[TestMethod]
public void ServiceImplementationTest()
{
var implementation = new Service(*** new MockDependency() ***);
bool result = implementation.DoSomething();
Assert.IsTrue(result);
}
Is this just a fact of life? Should I know all dependencies before writing the test? What happens when I want to write a new implementation with different dependencies, is it necessary to write a new test for each implementation, even though the correctness of the implementation hasn't changed?
Should you know your dependencies in advance for each test
Not necessarily, no.
What you are doing when you design "test first" is that you are exploring a possible API. So the following code
var implementation = new Service();
bool result = implementation.DoSomething();
Assert.IsTrue(result);
says, among other things, that the public API should allow you to create an instance of Service without knowing anything about its dependencies.
However, as I write the implementation, I realise I need a certain dependency, and so add a reference to that dependency in the constructor. To keep the test code compiling and failing, I then have to go back to the test, and modify it to create a fake implementation.
So notice two things here
This isn't a backwards compatible change...
Which means it isn't a refactoring
So part of your problem is that you are introducing the changes you want in a backwards incompatible way. If you were refactoring, you would have a step where your constructors would look something like
public Service() {
this(new IDependency() {
// default IDependencyImplementation here
});
}
Service(IDependency dependency) {
this.dependency = dependency;
}
At this point, you have two independent decisions
should Service(IDependency) be part of the public API
If so, then you start writing tests that force you to expose that constructor
should Service() be deprecated
and if it should, then you plan to delete the tests that depend on it when you remove Service() from the public API.
Note: that is a lot of unnecessary steps if you haven't shared/released the public API yet. Unless you are deliberately adopting the discipline that calibrated tests are immutable, it is usually more practical to hack the test to reflect the most recent draft of your API.
Don't forget to re-calibrate the test after you change it, though; any change to the test should necessarily trigger a refresh of the Red/Green cycle to ensure that the revised test is still measuring what you expect. You should never be publishing a test that you haven't calibrated.
A pattern that is sometimes useful is to separate the composition of the system under test from the automated checks
public void ServiceImplementationTest()
{
var implementation = new Service();
check(implementation);
}
void check(Service implementation) {
bool result = implementation.DoSomething();
Assert.IsTrue(result);
}
Another alternative is to write a spike - a draft of a possible implementation without tests, with purpose of exploring to better understand the constraints of the problem, which is thrown away when the learning exercise is completed.
Could you quickly describe what you mean by test calibration? Google isn't helping much. Is it making sure that tests fail when they are supposed to?
Yes: making sure that the fail when they are supposed so, making sure when they do fail that the messages reported are suitable, and of course making sure that they pass when they are supposed to.
I wrote some about the idea here.
im very curious for peoples opinions on what way a software side application that interfaces with hardware should be unit tested.
For example, the main class of the software application "Connection" would be constructing a handle to a USB device.
I want to test the "Connection" class base function, say "OpenConnection" that would attempt to connect to a USB hardware device.
so far I have constructed a MOCK hardware device, and have included in my connection class a compiler flag, so if its built in unit test mode, it will use a mock object, otherwish it will use the actual hardware interface.
See example below
class TConnection
{
public:
static TConnection* GetConnection();
static void Shutdown();
bool DidInitialise();
bool Write(uint8_t* _pu8_buffer);
bool Read(uint8_t* _pu8_buffer);
protected:
TConnection();
virtual ~TConnection();
bool init();
private:
static TConnection* mp_padConnection;
static bool mb_DidInitialise;
#ifdef _UNIT_TEST_BUILD
static mock_device* mp_handle;
#else
static device* mp_handle;
#endif
};
then in the source file I include something like
#include "connection.h"
#ifdef _UNIT_TEST_BUILD
mock_device* TConnection::mp_handle = nullptr;
#else
device* TConnection::mp_handle = nullptr;
#endif // _UNIT_TEST_BUILD
TConnection::TConnection()
{
...
init();
...
}
bool TConnection::init()
{
mp_handle = hid_open( _VENDOR_ID, _PRODUCT_ID, nullptr );
if (mp_hidHandle == nullptr) {
return false;
}
if (hid_set_nonblocking(mp_hidHandle, _DISABLE_NB) == _ERROR_CODE) {
return false;
}
return true;
}
The only thing I really dislike about my code is that my actual connection class contains test code. I would much prefer them to be separate.
Saying that, I also dont agree with having an entirely new mocked connection class written solely for the purpose of unit testing, it makes me feel like im just writting something designed to work as expected.
So I ask, what would be a better approach to testing such a class
Thank you in advance for your time and advice
You can avoid adding test code to your class by using dependency injection. Create an interface IDevice and make class Device implement that interface. Then, in class TConnection, use pointers to this interface instead of a member of type Device. Also create a helper method that allows you to set a new device, something like:
void setDevice(IDevice *device);
Now, for your production code simple use an instance of class Device, while in your test code use setDevice to swap implementation of device with a mock object. This mock object will be an instance of class MockDevice which will also implement interface IDevice. That way you can change the implementation in tests and use the mock class. Since you are using gtest already, I would suggest you do not write the mock class yourself but use C++ mocking framework gmock (which is fully compatible with gtest) instead. This way, you will also need to create a separate class but almost everything will be handled by the mocking framework. All you need to do is define mocked methods. Creation of an additional interface and mock class seems like overkill at first, but it definitely pays off in the long run. If you want to do any serious test-driving of code, learning to use interfaces, dependency injection and mock classes is essential. Check the documentation for more details:
https://github.com/google/googlemock/blob/master/googlemock/docs/CheatSheet.md
Personally I would have the mock as either a separate class, or part of the test code. To differentiate between mock and actual library I would do the changes in the build script, which I assume would be to include the test file (and mock) or linking to a library.
Creating a separate class is not wasted effort. It should behave as expected, but this can be simplified to the bare minimum needed for the test. A more interesting thing is to make the class generate error events, to make sure your code handles these events correctly. The alternative is sometimes to wait for the error to occur, which I wouldn't recommend.
Two highly recommended books on the subject:
Test-Driven Development for Embedded C: Part 2 gives a good overview of test doubles (mocks being one example) and how to use them.
Modern C++ Programming with Test-Driven Development: Less on test doubles, but more geared towards C++.
Suppose I have code like below. http_client is an external dependency (a 3rd party API) I don't have control over. transaction_handler is a class I control and would like to write unit tests for.
// 3rd party
class http_client
{
public:
std::string get(std::string url)
{
// makes an HTTP request and returns response content as string
// throws if status code is not 200
}
};
//code to be tested
enum class transaction_kind { sell, buy };
enum class status { ok, error };
class transaction_handler
{
private:
http_client client;
public:
status issue_transaction(transaction_kind action)
{
try
{
auto response =
client.get(std::string("http://fake.uri/") +
(action == transaction_kind::buy ? "buy" : "sell"));
return response == "OK" ? status::ok : status::error;
}
catch (const std::exception &)
{
return status::error;
}
}
};
Because http_client makes network calls I would like to be able to substitute it in my tests with a mock implementation which cuts off the network and allows for testing different conditions (ok/error/exception). Because transaction_handler is supposed to be internal I can modify it to make it testable but I wouldn't want to go over the border (i.e. I would like to avoid pointers or dynamic polymorphism if possible). Ideally I would like to use a kind of dependency injection where my tests would inject a mock http_client. I don't think I can/want to use a 'poor man's DI' where I would create an http_client implementation in the caller and pass it to the transaction_handler (by const reference? std::shared_ptr?) - because I don't control the http_client I would have to come up with an interface and -in the product code - I would have to wrap the http_client in a wrapper class that implements this interface and forwards the calls to the actual/wrapped http_client instance. In the test code I would create a mock implementation of that interface. The interface would have to be a pure abstract method which entails using runtime polymorphism which I wanted to avoid. Another option is to use templates. If I changed the transaction_handler class to look as follows:
template <typename T = http_client>
class transaction_handler
{
private:
T client;
public:
transaction_handler(const std::function<T()> &create) : client(create())
{}
status issue_transaction(transaction_kind action)
{
// same as above, omitted for brevity
}
}
I could now create a mock http_client class:
class http_client_mock
{
public:
std::string get(std::string url)
{
return std::string("OK");
}
};
and create the transaction_class object in my tests like this:
transaction_handler<http_client_mock> t(
[]() -> http_client_mock { return http_client_mock(); });
while I could use the following in my product code:
transaction_handler<> t1(
[]() -> http_client { return http_client(); });
While it seems to work and fullfill most of my requirements (even though I don't like the fact that the code instantiating transaction_handler need to be aware of the http_client type - maybe it can be somehow hidden as a factory class) - does it make sense at all? Or may be there are better ways of doing this kind of things? I spent a considerable amount of time looking for some simple DI patterns to make unit testing easier bud had hard time finding something that would suit my needs. Also, my background is mostly C so maybe I approach the problem from a wrong angle?
I'm maintaining a DI library, and your case is really interesting for me.
Mocking is about dynamic polymorphism or compile time mocking (at least when using C++). You pay 1 indirection to obtain ability to inject what you want (dependence only on 1 interface).
If you want to make code testable, the best way is using interfaces (pure virtual classes in C++ wich does not have interfaces) and inject dependencies only through constructor.
If you really want to avoid polymorphism (or you can't because of external API) you could still accept to have some code that is not fully testable.
Conventional way of doing things:
class ConcreteHttpClient : public virtual AbstractHttpClient { /*...*/}
class MockHttpClient : public virtual AbstractHttpClient{ /*...*/ }
You just choose what to inject based on needs ( I intentionally use "new" instead of showing some DI framwork at work).
Production code.
new TransactionHandler ( static_cast< AbstractService>( ConcreteService));
Unit testing the transaction handler
new TransactionHandler ( static_cast< AbstractService>( MockService));
If you later need to test some class using the transaction handler and the transaction handler implements a interface
class TransactionHandler: public virtual AbstractTransactionHandler { /*...*/}
You have just to create a TransactionHandlerMock inheriting from AbstractTransactionHandler
When you use interfaces the advantage is that you can use a Dependency Injection framework to avoid poor man's injection.
Compile time mocking.
What you proposed is a viable alternative, basically you assume a "static polymorphism" thanks to templates
Production code:
template <typename T = http_client>
class transaction_handler{/*...*/};
new transaction_handler( /*...*/ );
Unit test code:
using transaction_handler_mocked = transaction_handler< http_client_mock>;
new transaction_handler_mocked( /*...*/ );
However this has few Issues:
You are depending on "transaction_handler" type on each part of your production code so if you change it you have to recompile every file depending on it.
You can't inject a mocked handler as a mock itself, unless you change all classes depending on it to become templates accepting the handler.
Point number 2 means that basically in a complex project you have each class depending on each other, increasin compile times and forcing to whole recompiles of your project just for little changes
You are not using a Interface (pure virtual class) that means that you can still accidentally access fields or members of template parameters that were not intended to be accessed making your debuggin harder.
Other alternatives
Provide mock at link time, you don't have to mock a whole 3rd party library. Just the classes you are testing. Most IDEs are not thinked to work that way, but probably you can work around with some bash script or custom makefile. (In example, I do that for testing code dependent on C functions, in particular OpenGL)
Wrap interesting functionalities of libraries you want to test behind a class implementing a pure virtual class (don't know why you want to avoid it). You have good chances that you don't need to wrap all methods and you'll end with a smaller API to test against (just wrap parts you need to use, don't start up front wrapping the whole library)
Use a mocking framework, and possibly a dependency Injection framework.
Just write test routines that match whichever exports of http_client you're using. You're source will be linked in preference to any lib.
I think it's a good idea to mock it. Since http_client is an external dependency, I chose Typemock Isolator++ to handle with it. Look at the code below:
TEST_METHOD(FakeHttpClient)
{
//Arrange
string okStr = "OK";
http_client* mock_client = FAKE_ALL<http_client>();
WHEN_CALLED(mock_client->get(ANY_VAL(std::string))).Return(&okStr);
//Act
transaction_handler my_handler;
status result = my_handler.issue_transaction(transaction_kind::buy);
//Assert
Assert::AreEqual((int)status::ok, (int)result);
}
Method FAKE_ALL<> allows me to set the behavior of all http_client instances, so no injection needed. Simple API, the code looks accurate, and you don't need to change the production code.
Hope it helps!
I often face the problem that mock objects need to be brought in a certain state before the "interesting" part of a test can start.
For example, let's say I want to test the following class:
struct ToTest
{
virtual void onEnable();
virtual void doAction();
};
Therefore, I create the following mock class:
struct Mock : ToTest
{
MOCK_METHOD0(onEnable, void());
MOCK_METHOD0(doAction, void());
};
The first test is that onEnable is called when the system that uses a ToTest object is enabled:
TEST(SomeTest, OnEnable)
{
Mock mock;
// register mock somehow
// interesting part of the test
EXPECT_CALL(mock, onEnable());
EnableSystem();
}
So far, so good. The second test is that doAction is called when the system performs an action and is enabled. Therefore, the system should be enabled before the interesting part of the test can start:
TEST(SomeTest, DoActionWhenEnabled)
{
Mock mock;
// register mock somehow
// initialize system
EnableSystem();
// interesting part of the test
EXPECT_CALL(mock, doAction());
DoSomeAction();
}
This works but gives an annoying warning about an uninteresting call to onEnable. There seem to be two common fixes of this problem:
Using NiceMock<Mock> to suppress all such warnings; and
Add an EXPECT_CALL(mock, onEnable()) statement.
I don't want to use the first method since there might be other uninteresting calls that really should not happen. I also don't like the second method since I already tested (in the first test) that onEnable is called when the system is enabled; hence, I don't want to repeat that expectation in all tests that work on enabled systems.
What I would like to be able to do is say that all mock calls up to a certain point should be completely ignored. In this example, I want expectations to be only checked starting from the "interesting part of the test" comment.
Is there a way to accomplish this using Google Mock?
The annoying thing is that the necessary functions are there: gmock/gmock-spec-builders.h defines Mock::AllowUninterestingCalls and others to control the generation of warnings for a specific mock object. Using these functions, it should be possible to temporarily disable warnings about uninteresting calls.
That catch is, however, that these functions are private. The good thing is that class Mock has some template friends (e.g., NiceMock) that can be abused. So I created the following workaround:
namespace testing
{
// HACK: NiceMock<> is a friend of Mock so we specialize it here to a type that
// is never used to be able to temporarily make a mock nice. If this feature
// would just be supported, we wouldn't need this hack...
template<>
struct NiceMock<void>
{
static void allow(const void* mock)
{
Mock::AllowUninterestingCalls(mock);
}
static void warn(const void* mock)
{
Mock::WarnUninterestingCalls(mock);
}
static void fail(const void* mock)
{
Mock::FailUninterestingCalls(mock);
}
};
typedef NiceMock<void> UninterestingCalls;
}
This lets me access the private functions through the UninterestingCalls typedef.
The flexibility you're looking for is not possible in gmock, by design. From the gmock Cookbook (emphasis mine):
[...] you should be very cautious about when to use naggy or strict mocks, as they tend to make tests more brittle and harder to maintain. When you refactor your code without changing its externally visible behavior, ideally you should't need to update any tests. If your code interacts with a naggy mock, however, you may start to get spammed with warnings as the result of your change. Worse, if your code interacts with a strict mock, your tests may start to fail and you'll be forced to fix them. Our general recommendation is to use nice mocks (not yet the default) most of the time, use naggy mocks (the current default) when developing or debugging tests, and use strict mocks only as the last resort.
Unfortunately, this is an issue that we, and many other developers, have encountered. In his book, Modern C++ Programming with Test-Driven Development, Jeff Langr writes (Chapter 5, on Test Doubles):
What about the test design? We split one test into two when we changed from a hand-rolled mock solution to one using Google Mock. If we expressed everything in a single test, that one test could set up the expectations to cover all three significant events. That’s an easy fix, but we’d end up with a cluttered test.
[...]
By using NiceMock, we take on a small risk. If the code later somehow changes to invoke another method on the [...] interface, our tests aren’t going to know about it. You should use NiceMock when you need it, not habitually. Seek to fix your design if you seem to require it often.
You might be better off using a different mock class for your second test.
class MockOnAction : public ToTest {
// This is a non-mocked function that does nothing
virtual void onEnable() {}
// Mocked function
MOCK_METHOD0(doAction, void());
}
In order for this test to work, you can have onEnable do nothing (as shown above). Or it can do something special like calling the base class or doing some other logic.
virtual void onEnable() {
// You could call the base class version of this function
ToTest::onEnable();
// or hardcode some other logic
// isEnabled = true;
}
I'm attempting to help design some unit tests around controllers in a Qt C++ application.
To be frank, I have two large drawbacks. One, my testing background is heavily based on .NET projects, so my knowledge of best practice in the c++ world is slim at best. Two, the designer of the application I am looking at did not architect the code with unit testing in mind.
One specific point point, I'm looking at a controller class that includes boost/filesystem/operations.hpp. The controller constructor goes on to check directory existence and create directories using functions from the boost filesystem code.
Is there any way to overload or mock this behavior? I'm used to setting up an IoC container or at least dependency injected constructors in .NET, and then being able to pass mock objects in the unit test code. I'm not sure how a templated header file would work with that concept, though, or if it is even typical practice in c++.
Currently, I have no flexibility to suggest code changes, as there is a release build coming up this week. But after that, if there are some simple code changes that could improve testability, that is definitely an option. Ideally, there would be a way to overload the filesystem functions in the unit test framework as is, though.
We ended up creating a generic file system wrapper that calls Boost filesystem and accepting it as a parameter to our class constructors so we could send in mock versions at unit test time.
I understand the thought to not mock this, but I think there is value in fast unit tests for our CI environment to run at check in time as well as tests that actually hit the file system.
If you think about it, the only reason why you need to inject an instance of, suppose, IFilesystem into your classes, is to mock it in tests. No part of your non-test codebase gonna use anything but the real filesystem anyway, so it is safe to assume that you can inject not an object but a type into your classes and use them freely within your codebase without type clashes.
So, suppose you have class of interest
struct BuildTree
{
BuildTree(std::string_view dirname) { /*...*/ }
bool has_changed()
{
// iterates through files in directory
// recurs into directories
// checks the modification date against lastModified_
// uses boost::filesystem::last_write_time(), etc.
}
private:
boost::filesystem::file_time_type lastModified_;
};
Now, you could inject a type into the class. This type will be a class with a bunch of static methods. There will be a RealFilesystem type that will redirect to the boost::filesystem methods, and there will be a SpyFilesystem.
template <class Filesystem>
struct BuildTree
{
// ...
bool has_changed()
{
// uses Filesystem::last_write_time(), etc.
}
private:
typename Filesystem::file_time_type lastModified_;
};
The SpyFilesystem will resemble a PIMPL idiom, in that the static methods will redirect calls to the actual implementation.
struct SpyFilesystemImpl;
struct SpyFilesystem
{
using file_time_type = typename SpyFilesystemImpl::file_time_type;
static file_time_type last_time_write(std::string_view filename)
{
return instance.last_time_write(filename);
}
// ... more methods
static SpyFilesystemImpl instance;
};
SpyFilesystemImpl SpyFilesystem::instance{};
// No warranty of completeness provided
struct SpyFilesystemImpl
{
using file_time_type = std::chrono::system_clock::time_point;
void create_directory(std::string_view path) { /*...*/ }
void touch(std::string_view filename)
{
++lastModified_[filename];
}
file_time_type last_time_write(std::string_view filename)
{
return std::chrono::system_clock::time_point{lastModified_[filename]};
}
private:
std::unordered_map<std::string, std::chrono::seconds> lastModified_;
};
Finally, inside each of your tests you would prepare an instance of SpyFilesystemImpl, assign it to the SpyFilesystem::instance and then instantiate your class with SpyFilesystem. like so
// ... our SpyFilesystem and friends ...
// Google Test framework
TEST(BuildTree, PickUpChanges)
{
SpyFilesystemImpl fs{};
fs.create_directory("foo");
fs.touch("foo/bar.txt");
SpyFilesystem::instance = fs;
BuildTree<SpyFilesystem> tree("foo");
EXPECT_FALSE(tree.has_changed());
SpyFilesystem::instance.touch("foo/bar.txt");
EXPECT_TRUE(tree.has_changed());
}
This approach has an advantage that there will be no run-time overhead in the resulting binary (provided that optimizations are enabled). However, it requires more boilerplate code, which might be a problem.
It seems reasonable to me to consider boost::filesystem as an
extension of the standard libraries. So you mock it (or not) in
exactly the same way you mock something like std::istream.
(Generally, of course, you don't mock it, but rather your test
framework provides the necessary environment: the files you need
to read with std::istream, the directories, etc. for
boost::filesystem.)