I have several functions to unit test
public class MyClass {
public DataObj FuncX (int input1, int input2) {
if( input1 < 0 && input2 > 0 ){
return Func01();
}
else if(input1 < 2 && input2 < 0) {
return Func02();
}
....
return Func0N();
}
}
I already have many unit tests for functions like Func01(), Func02(), but I don't know how to unit test FuncX().
(Note: Func01() and other functions are not virtual)
Just like testing private methods, you can't actually test functions inside a method. You should expect to have result X from a function Y no matter what things are happening inside of it.
There is an unspoken contract between the function you are testing and the functions that it's calling that is ensuring everything should work as expected. If internal function don't work as it's supposed to be, the external one won't work either, but the reason of test failure should not be the concern of your test in that case.
Create tests for internal functions and then for the one that's calling them, this way you'll be able to find the cause of test failure easily.
Your example seems to suggest the following options:
Do a 'full' test of FuncX, including the test of the functions called - This is undesirable because you have already tested Func01 etc., and a full test of FuncX would then duplicate all these tests.
Do an isolated test of FuncX, by (somehow) using doubles for the called functions Func01 etc. This is difficult, because the called functions are not virtual.
There is, however, another option: When unit-testing a function with dependencies, you only have to do something about a dependency if it causes you trouble. If it doesn't cause trouble, you just can live with the dependency. For example, if some code depends on the programming language's mathematical library (for example the sin() function), this is also a dependency - but this dependency is almost never replaced by doubles.
Good reasons to use doubles are if the dependencies cause non-deterministic behaviour, take long to compute, are buggy or not implemented yet or make certain test scenarios hard to perform - to name some important ones.
But, if your dependencies do not cause any trouble - just perform the unit tests with them. In your example, just live with the fact that FuncX calls Func01 etc. As long as you can from the results easily determine whether FuncX has worked properly (i.e. which decision FuncX has taken), you can test FuncX as well as if you had doubled Func01 and the others.
Related
Suppose we have a function add() as below:
void add(int a, int b) {
int sum=a+b;
cout<<sum;
sendSumToStorage(sum);
}
This simple function adds to input values, prints the sum to the console and also sends it to some external storage (say, a file). This is how we ideally want it in the application (meaning, we don't want it to return anything).
For purposes of unit testing, is it valid (from a design perspective) if we modify the function signature so that it returns the sum? We could then have a test like:
bool checkAdd() {
int res=add(3, 4);
if(res==7) return true;
else return false;
}
Better yet, is this (returning a value) the only way we could unit test it? Is there some valid way in which we could unit test the add() function without changing the function signature?
A function like the one in your example is considered really bad practice.
Why do I say this?
Well, you have a method called add which adds two numbers AND calls something else. Basically your method doesn't do one thing, but two, which violates the Single Responsibility Principle.
This makes things much harder to test because you can't test just the add method in isolation.
So you would separate that into two methods with good names which reflect what they do and test them separately.
If you don't want to have issues with state between your methods, you will have to start returning results where it makes sense.
Ignoring the fact that this example has a bad desing.
For cases like this, when you want to check some internal behaviour instead of API you should rather try using some testing libraries like gtest and gmock.
It allows you to describe more sophisticated expectations than just function result.
For example you can set expectation that some method will be called during code execution using EXPECT_CALL macro.
More details here:
https://github.com/google/googletest/blob/master/googlemock/docs/ForDummies.md
Answering your question, it's always a bad practice to modify any part of tested code for the purpose of testing. In that case you are not longer testing production code. As it was suggested before it's better to split the functionality into smaller parts and test them isolated.
Changing the design of code to improve testability is very common and generally considered as a good practice. Obviously, not all such changes are necessarily real improvements - sometimes better solutions exist.
In your case, the code is difficult to test because it combines computations (the addition) with interactions with depended-on components (output and storing data). In your case (as Andrei has pointed out) the function also violates the SRP, but mixing computations and interactions generally makes testing more difficult, even in cases where the SRP is not violated.
I understand, you would change the function such that it will return the computed value in addition to printing it and writing it to the storage. This will allow you to test the computational part of the function. The function will then, however, only be partially tested. And, the purpose of the function will be further obfuscated.
If instead you split the function into one function doing the computation and one doing the interactions, you can thoroughly test the first with unit-testing, and use integration-testing for the other. Again, the usefulness of this approach is independent of whether the code violates the SRP or not.
I've been doing some unit testing and just getting into the topic as a whole.
I stumbled upon the following scenario, suppose I have a class like this:
class A{
public B mehtod_1(B b){
b = method_2(b);
b = method_3(b);
b += 1;
return b;
}
public B method_2(B b){
// do something to B without external dependency
return B;
}
public B method_3(B b){
// do something else to B without external dependency
return B;
}
}
I can write tests for method_2 and method_3 without a problem, do different tests by configuring B in different ways and asserting the expected transformation on B after the call, those methods are atomic.
So my question is:
If I was to test method_1 in an atomic way I would have to mock the calls to method_2 and method_3 since if I would actually call these methods I would not test method_1 in an atomic manor.
In the latter case is method_2 was broken then the tests for method_1 and method_2 would break, and that would be misleading. If I'd mock the method_2 call inside the method_1 test, only the method_2 test would fail, giving a clearer indication of where the error is (namely somewhere in the business logic of method_1 given all other invoked methods worked as expected).
Did I understand the concept here correctly?
On the other hand it is correct, if both tests fail, since in the real world, method_1 cannot work without method_2 working.
My gut would say atomicity of tests is what is desired, meaning the first solution where there is one test for method_1, for every possible outcome of method_2 and method_3 (statically mocked).
Is there a "correct"/common/best practice way?
Immediate answer: in case we are talking Java here; and partial mocking is really of interest to you, you can look into using Mockito's spy concept.
But beyond that: you are getting unit testing wrong. What you call atomicity; I call worrying about implementation details. But it shouldn't matter "what exactly" that "method under test" actually does. You want to test the what, not the how.
Meaning: if that method has to call some other method(s) (that work fine in your unit test environment; without mocking); then there is no need thinking about mocking them!
You see: you care about the contract of each of your methods. That contract is what you want to test: given these input parameters, I expect that result/side effect/exception ...
Nonetheless, the fact that you have multiple public methods; and that they somehow depend on each other might be an indication of a design problem (as in: does it make sense that they are all public; is there some abstraction hiding in your interface that you should better express in other ways?). But that can only be decided given real code; real context.
I'm looking to better understand I should test functions that have many substeps or subfunctions.
Let's say I have the functions
// Modify the state of class somehow
public void DoSomething(){
DoSomethingA();
DoSomethingB();
DoSomethingC();
}
Every function here is public. Each subfunction has 2 paths. So to test every path for DoSomething() I'd have 2*2*2 = 8 tests. By writing 8 tests for DoSomething() I will have indirectly tested the subfunctions too.
So should I be testing like this, or instead write unit tests for each of the subfunctions and then only write 1 test case that measures the final state of the class after DoSomething() and ignore all the possible paths? A total of 2+2+2+1 = 7 tests. But is it bad then that the DoSomething() test case will depend on the other unit test cases to have complete coverage?
There appears to be a very prevalent religious belief that testing should be unit testing. While I do not intend to underestimate the usefulness of unit testing, I would like to point out that it is just one possible flavor of testing, and its extensive (or even exclusive) use is indicative of people (or environments) that are somewhat insecure about what they are doing.
In my experience knowledge of the inner workings of a system is useful as a hint for testing, but not as an instrument for testing. Therefore, black box testing is far more useful in most cases, though that's admittedly in part because I do not happen to be insecure about what I am doing. (And that is in turn because I use assertions extensively, so essentially all of my code is constantly testing itself.)
Without knowing the specifics of your case, I would say that in general, the fact that DoSomething() works by invoking DoSomethingA() and then DoSomethingB() and then DoSomethingC() is an implementation detail that your black-box test should best be unaware of. So, I would definitely not test that DoSomething() invokes DoSomethingA(), DoSomethingB(), and DoSomethingC(), I would only test to make sure that it returns the right results, and using the knowledge that it does in fact invoke those three functions as a hint I would implement precisely those 7 tests that you were planning to use.
On the other hand, it should be noted that if DoSomethingA() and DoSomethingB() and DoSomethingC() are also public functions, then you should also test them individually, too.
Definitely test every subfunction seperately (because they're public).
It would help you find the problem if one pops up.
If DoSomething only uses other functions, I wouldn't bother writing additional tests for it. If it has some other logic, I would test it, but assume all functions inside work properly (if they're in a different class, mock them).
The point is finding what the function does that is not covered in other tests and testing that.
Indirect testing should be avoided. You should write unit tests for each function explicitly. After that You should mock submethods and test your main function. For example :
You have a method which inserts a user to DB and method is like this :
void InsertUser(User user){
var exists = SomeExternal.UserExists(user);
if(exists)
throw new Exception("bla bla bla");
//Insert codes here
}
If you want to test InsertUser function, you should mock external/sub/nested methods and test behaviour of InsertUser function.
This example creates two tests: 1 - "When user exists then Should throw Exception" 2 - "When user does not exist then Should insert user"
I am into a project where we have a large codebase and currently it has no unit tests framework at all.The code we are working on would be finally run on a box which acts as a switch/router/firewall.
So I am working on a piece of code which needs to be unit-tested using Gtest.
the problem I have with this is mocking the variables in order to test the function itself.
For eg I have a function which uses like 4 pointers to different objects and uses couple of global variables .In order to test different paths in code I need to initialize almost the entire state machien/values of dependent variables.
Adding to the complexity as it is true in a large codebase this function/method I have written uses a bunch of other routines/methods which needs to be tested as well. Each of those needs to be uni-tested as well with each of them having their own dependencies.
I am not sure whether I am approching the problem right or is it the case that gtest may not be the right tool for testing such large code-base.
If anyone has experience with say testing say a call-stack say
function A {
code
code
function B
code
code
function C
code
}
function B
{
function D
code
function E
}
function C{
code
function F
function G
code
}
something like this.How do I test all these function A-F ?? What is a good strategy ??
First thing is refactoring the code so that testable pieces are isolated. In particular, this means removing the access to globals. Example:
int global;
int function() {
int r = foo();
global += r / 2;
bar(r);
return 42;
}
Removing the global object means converting it to an input parameter:
int real_function(int* target) {
assert(target);
int r = foo();
*target += r / 2;
bar(r);
return 42;
}
Then of course the remaining code will stop compiling, so you add a backward-compatibility cludge:
int global_bar;
// #deprecated, use real_function() directly
int function() {
return real_function(&global_bar);
}
Using that, you step up the call chain, extracting the dependencies and hopefully one day removing the last call to the variant that requires the globals. In the meantime, you can write tests for the functions that don't depend on global stuff any longer. Note that for C++, you would use references instead of pointers and probably pass the required external objects to the class constructor. This is also called dependency injection, make sure to research that term for a thorough understanding.
Another way to test functions touching globals is to use the setup function of the test to reset the global to a known state. This still requires linking in the global though, which may prove difficult. And not using globals may make the codebase much better in the first place, so accepting that also sends the wrong message.
Ulrich Eckhardt essentially says, "You need to get rid of globals to make easily testable code". But you really should go further.
For any global function you want to test you should look at
The globals it accesses.
The parameters it uses.
the functions it calls.
Then consider:
Converting the functions it calls to calls to one or more interfaces, and pass those as parameters.
Converting the globals to parameters, or function calls on an interface.
If your function is a function on an object rather than a global function, you can consider additionally:
making the globals member variables, and passing them to the constructor
making the functions it calls virtual member functions
The final thing I'd consider for making a function testable, is whether it belongs in a class.
Once all these are addressed you can usually mock the required bits easily. If you're using gtest, you may be able to use gmock to make this easier. (I've used gmock with gtest before and its pretty painless.)
(And yes, I've taken this approach on large code-bases before... its usually pretty painful to start with, but once you get used to it and the code starts to get more testable - things will improve.)
I've been reading that static methods tend to be avoided when using TDD because they tend to be hard to mock. I find though, that the easiest thing to unit test is a static method that has simple functionality. Don't have to instantiate any classes, encourages methods that a simple, do one thing, are "standalone" etc.
Can someone explain this discrepancy between TDD best practices and pragmatic ease?
thanks,
A
A static method is easy to test, but something that directly calls a static method generally is not easy to test independent of the static method it depends on. With a non-static method you can use a stub/mock/fake instance to ease testing, but if the code you're testing calls static methods it's effectively "hard-wired" to that static method.
The answer to the asked question is, in my opinion "Object Oriented seems to be all that TDD people think about."
Why? I don't know. Maybe they are all Java programmers who've been infected with the disease of making everything rely on six indirection layers, dependency injection and interface adapters.
Java programmers seem to love to make everything difficult up front in order to "save time later."
I advise applying some Agile principles to your TDD: If it isn't causing a problem then don't fix it. Don't over design.
In practice I find that if the static methods are tested well first then they are not going to be the cause of bugs in their callers.
If the static methods execute quickly then they don't need a mock.
If the static methods work with stuff from outside the program, then you might need a mock method. In this case you'd need to be able to simulate many different kinds of function behavior.
If you do need to mock a static method remember that there are ways to do it outside of OO programming.
For example, you can write scripts to process your source code into a test form that calls your mock function. You could link different object files that have different versions of the function into the test programs. You could use linker tricks to override the function definition (if it didn't get inlined). I am sure there are some more tricks I haven't listed here.
It's easy to test the static method. The problem is that there is no way to isolate your other code from that static method when testing the other code. The calling code is tightly-coupled to the static code.
A reference to a static method cannot be mocked by many mocking frameworks nor can it be overridden.
If you have a class that is making lots of static calls, then to test it you have to configure the global state of the application for all of those static calls - so maintenance becomes a nightmare. And if your test fails, then you don't know which bit of code caused the failure.
Getting this wrong, is one of the reasons that many developers think TDD is nonsense. They put in a huge maintenance effort for test results that only vaguely indicate what went wrong. If they'd only reduced the coupling between their units of code, then maintenance would be trivial and the test results specific.
That advice is true for the most part.. but not always. My comments are not C++ specific..
writing tests for static methods (which are pure/stateless functions): i.e. the work off the inputs to produce a consistent result. e.g. Add below - will always give the same value given a particular set of inputs. There is no problem in writing tests for these or code that calls such pure static methods.
writing tests for static methods which consume static state : e.g. GetAddCount() below. Calling it in multiple tests can yield different values. Hence one test can potentially harm the execution of another test - tests need to be independent. So now we need to introduce a method to reset static state such that each test can start from a clean slate (e.g. something like ResetCount()).
Writing tests for code that accesses static methods but no source-code access to the dependency: Once again depends on the properties of the static methods themselves. However if they are gnarly, you have a difficult dependency. If the dependency is an object, then you could add a setter to the dependent type and set/inject a fake object for your tests. When the dependency is static, you may need a sizable refactoring before you can get tests running reliably. (e.g. Add an object middle-man dependency that delegates to the static method. Now plugin a fake middle-man for your tests)
Lets take an example
public class MyStaticClass
{
static int __count = 0;
public static int GetAddCount()
{ return ++__count; }
public static int Add(int operand1, int operand2)
{ return operand1 + operand2; }
// needed for testability
internal static void ResetCount()
{
__count = 0;
}
}
...
//test1
MyStaticClass.Add(2,3); // => 5
MyStaticClass.GetAddCount(); // => 1
// test2
MyStaticClass.Add(2,3); // => 5
//MyStaticClass.ResetCount(); // needed for tests
MyStaticClass.GetAddCount(); // => unless Reset is done, it can differ from 1