I am into a project where we have a large codebase and currently it has no unit tests framework at all.The code we are working on would be finally run on a box which acts as a switch/router/firewall.
So I am working on a piece of code which needs to be unit-tested using Gtest.
the problem I have with this is mocking the variables in order to test the function itself.
For eg I have a function which uses like 4 pointers to different objects and uses couple of global variables .In order to test different paths in code I need to initialize almost the entire state machien/values of dependent variables.
Adding to the complexity as it is true in a large codebase this function/method I have written uses a bunch of other routines/methods which needs to be tested as well. Each of those needs to be uni-tested as well with each of them having their own dependencies.
I am not sure whether I am approching the problem right or is it the case that gtest may not be the right tool for testing such large code-base.
If anyone has experience with say testing say a call-stack say
function A {
code
code
function B
code
code
function C
code
}
function B
{
function D
code
function E
}
function C{
code
function F
function G
code
}
something like this.How do I test all these function A-F ?? What is a good strategy ??
First thing is refactoring the code so that testable pieces are isolated. In particular, this means removing the access to globals. Example:
int global;
int function() {
int r = foo();
global += r / 2;
bar(r);
return 42;
}
Removing the global object means converting it to an input parameter:
int real_function(int* target) {
assert(target);
int r = foo();
*target += r / 2;
bar(r);
return 42;
}
Then of course the remaining code will stop compiling, so you add a backward-compatibility cludge:
int global_bar;
// #deprecated, use real_function() directly
int function() {
return real_function(&global_bar);
}
Using that, you step up the call chain, extracting the dependencies and hopefully one day removing the last call to the variant that requires the globals. In the meantime, you can write tests for the functions that don't depend on global stuff any longer. Note that for C++, you would use references instead of pointers and probably pass the required external objects to the class constructor. This is also called dependency injection, make sure to research that term for a thorough understanding.
Another way to test functions touching globals is to use the setup function of the test to reset the global to a known state. This still requires linking in the global though, which may prove difficult. And not using globals may make the codebase much better in the first place, so accepting that also sends the wrong message.
Ulrich Eckhardt essentially says, "You need to get rid of globals to make easily testable code". But you really should go further.
For any global function you want to test you should look at
The globals it accesses.
The parameters it uses.
the functions it calls.
Then consider:
Converting the functions it calls to calls to one or more interfaces, and pass those as parameters.
Converting the globals to parameters, or function calls on an interface.
If your function is a function on an object rather than a global function, you can consider additionally:
making the globals member variables, and passing them to the constructor
making the functions it calls virtual member functions
The final thing I'd consider for making a function testable, is whether it belongs in a class.
Once all these are addressed you can usually mock the required bits easily. If you're using gtest, you may be able to use gmock to make this easier. (I've used gmock with gtest before and its pretty painless.)
(And yes, I've taken this approach on large code-bases before... its usually pretty painful to start with, but once you get used to it and the code starts to get more testable - things will improve.)
Related
In my past experience of testing C code, the function stub is pretty much mandatory. In my safety-critical line of work, I am usually required to test everything - even abstractive functions which simply call a build-specific implementation argument for argument, return for return.
I have searched high and low for ways to, in Rust, stub functions external to the module under test, but cannot find a solution.
There are several libraries for trait-mocking, but what little I have read about this topic suggests that it is not what I am looking for.
The other suggestion is that functions I test which call external functions have those functions passed in, allowing the test simply to pass the desired pseudostub function. This seems to be very inflexible in terms of the data passing in and out of the pseudostub and causes one's code to be polluted with function-reference arguments at every level - very undesirable when the function under test would never call anything except one operational function or the stub. You are writing operational code to fit in with the limitations of the testing system.
This seems so very basic. Surely there is a way to stub external functions with Rust and Cargo?
You can try and use mock crates like mockall, which I think is the more complete out there, but still might need some time getting used to.
Without a mock crate, I would suggest mocking the traits/structs in an other, and then bringing then into scope with the #[cfg(test)] attribute. Of course this would make it mandatory for you to annotate the production use statement with `#[cfg(not(test))]. for example:
If you're using an external struct ExternalStruct from external-crate with a method external_method you would have something like:
file real_code.rs
#[cfg(not(test))]
use external-crate::ExternalStruct;
#[cfg(test)]
use test_mocks::ExternalStruct;
fn my_function() {
//...
ExternalTrait::external_method();
}
#[test]
fn test_my_function(){
my_function();
}
file test_mocks.rs:
pub Struct ExternalStruct {}
impl ExternalStruct {
pub fn external_method() {
//desired mock behaviour
}
}
On running your test the ExternalStruct from the test_mocks will be used and the real dependency otherwise.
Suppose we have a function add() as below:
void add(int a, int b) {
int sum=a+b;
cout<<sum;
sendSumToStorage(sum);
}
This simple function adds to input values, prints the sum to the console and also sends it to some external storage (say, a file). This is how we ideally want it in the application (meaning, we don't want it to return anything).
For purposes of unit testing, is it valid (from a design perspective) if we modify the function signature so that it returns the sum? We could then have a test like:
bool checkAdd() {
int res=add(3, 4);
if(res==7) return true;
else return false;
}
Better yet, is this (returning a value) the only way we could unit test it? Is there some valid way in which we could unit test the add() function without changing the function signature?
A function like the one in your example is considered really bad practice.
Why do I say this?
Well, you have a method called add which adds two numbers AND calls something else. Basically your method doesn't do one thing, but two, which violates the Single Responsibility Principle.
This makes things much harder to test because you can't test just the add method in isolation.
So you would separate that into two methods with good names which reflect what they do and test them separately.
If you don't want to have issues with state between your methods, you will have to start returning results where it makes sense.
Ignoring the fact that this example has a bad desing.
For cases like this, when you want to check some internal behaviour instead of API you should rather try using some testing libraries like gtest and gmock.
It allows you to describe more sophisticated expectations than just function result.
For example you can set expectation that some method will be called during code execution using EXPECT_CALL macro.
More details here:
https://github.com/google/googletest/blob/master/googlemock/docs/ForDummies.md
Answering your question, it's always a bad practice to modify any part of tested code for the purpose of testing. In that case you are not longer testing production code. As it was suggested before it's better to split the functionality into smaller parts and test them isolated.
Changing the design of code to improve testability is very common and generally considered as a good practice. Obviously, not all such changes are necessarily real improvements - sometimes better solutions exist.
In your case, the code is difficult to test because it combines computations (the addition) with interactions with depended-on components (output and storing data). In your case (as Andrei has pointed out) the function also violates the SRP, but mixing computations and interactions generally makes testing more difficult, even in cases where the SRP is not violated.
I understand, you would change the function such that it will return the computed value in addition to printing it and writing it to the storage. This will allow you to test the computational part of the function. The function will then, however, only be partially tested. And, the purpose of the function will be further obfuscated.
If instead you split the function into one function doing the computation and one doing the interactions, you can thoroughly test the first with unit-testing, and use integration-testing for the other. Again, the usefulness of this approach is independent of whether the code violates the SRP or not.
I have several functions to unit test
public class MyClass {
public DataObj FuncX (int input1, int input2) {
if( input1 < 0 && input2 > 0 ){
return Func01();
}
else if(input1 < 2 && input2 < 0) {
return Func02();
}
....
return Func0N();
}
}
I already have many unit tests for functions like Func01(), Func02(), but I don't know how to unit test FuncX().
(Note: Func01() and other functions are not virtual)
Just like testing private methods, you can't actually test functions inside a method. You should expect to have result X from a function Y no matter what things are happening inside of it.
There is an unspoken contract between the function you are testing and the functions that it's calling that is ensuring everything should work as expected. If internal function don't work as it's supposed to be, the external one won't work either, but the reason of test failure should not be the concern of your test in that case.
Create tests for internal functions and then for the one that's calling them, this way you'll be able to find the cause of test failure easily.
Your example seems to suggest the following options:
Do a 'full' test of FuncX, including the test of the functions called - This is undesirable because you have already tested Func01 etc., and a full test of FuncX would then duplicate all these tests.
Do an isolated test of FuncX, by (somehow) using doubles for the called functions Func01 etc. This is difficult, because the called functions are not virtual.
There is, however, another option: When unit-testing a function with dependencies, you only have to do something about a dependency if it causes you trouble. If it doesn't cause trouble, you just can live with the dependency. For example, if some code depends on the programming language's mathematical library (for example the sin() function), this is also a dependency - but this dependency is almost never replaced by doubles.
Good reasons to use doubles are if the dependencies cause non-deterministic behaviour, take long to compute, are buggy or not implemented yet or make certain test scenarios hard to perform - to name some important ones.
But, if your dependencies do not cause any trouble - just perform the unit tests with them. In your example, just live with the fact that FuncX calls Func01 etc. As long as you can from the results easily determine whether FuncX has worked properly (i.e. which decision FuncX has taken), you can test FuncX as well as if you had doubled Func01 and the others.
Sometimes I need to get variable variable, that can be accessed from any place of code. But I often hear that global variables are bad. What is the best practice in D for such case?
Now my code look like:
string roothtml;
static this()
{
roothtml = buildPath(getcwd, "html");
}
void main()
{
//
}
Is it's good or bad practice?
Global variables are problematic for a few reasons.
It's harder to track where variables are coming from when you're reading. This makes it harder to understand a function using globals.
It's harder to track where a variable is used. This makes it harder to modify how your global variables should be used.
It's more awkward to inject test data and test stubs.
Test state will spill over to other tests.
__gshared globals require locking or immutability.
Thread-local globals are thread-local, so you can't propagate writes to all threads.
Any globals require you to think about whether you want it thread-local or __gshared.
If you need to convert your single-tenant application into a multi-tenant one, that will be painful if you're using global variables. And it's more common than you probably suspect.
You have to be careful not to build with -unittest when running your application so you don't trash global state initialized in a static constructor.
On the plus side, it's convenient that you don't have to pass global state everywhere. You don't have to use method to method object refactorings as often. You don't have to bring in a dependency injection system. It's very convenient when it's not biting you.
It depends on what you really mean by 'global'. In the example above, I'd say its fine.
You appear to be showing a main module, which probably shouldn't be imported
by anything. In other words, it isn't really global, it is local to the main
module. It really isn't so different from
class Main {
private string _roothtml;
static this() { _roothtml = buildPath(getcwd, "html"); }
void run() { }
}
Even if it isn't really your main, D's
module system offers protections of its
own. Just stick a private on roothtml to encapsulate it within the module
(it wouldn't hurt to do this in your main module anyways, just to be clear).
A pattern like this is widely employed in the source code of git. Rather than
having a single main module that invokes a function for a given command, you
have many main functions -- one for each top-level command.
For example, take a look at
upload-pack.c.
See those variables declared at the top of the source file?
Would the code have been any clearer or safer if they were wrapped in a class in
typical OOP style or of explicitly passed to each function in a more purely
functional style?
Each source file acts as a unit of encapsulation for a given command. This style is not always appropriate, but in the case of a program that can be thought of as a set of distinct commands, it can be cleaner than the alternatives.
Ultimately, the answer will be specific to the context, your given project, and
your personal style. Generally speaking, cross-module globals are something to
be looked on with suspicion, but module-level variables can sometimes be cleaner
than the alternatives.
I've been reading that static methods tend to be avoided when using TDD because they tend to be hard to mock. I find though, that the easiest thing to unit test is a static method that has simple functionality. Don't have to instantiate any classes, encourages methods that a simple, do one thing, are "standalone" etc.
Can someone explain this discrepancy between TDD best practices and pragmatic ease?
thanks,
A
A static method is easy to test, but something that directly calls a static method generally is not easy to test independent of the static method it depends on. With a non-static method you can use a stub/mock/fake instance to ease testing, but if the code you're testing calls static methods it's effectively "hard-wired" to that static method.
The answer to the asked question is, in my opinion "Object Oriented seems to be all that TDD people think about."
Why? I don't know. Maybe they are all Java programmers who've been infected with the disease of making everything rely on six indirection layers, dependency injection and interface adapters.
Java programmers seem to love to make everything difficult up front in order to "save time later."
I advise applying some Agile principles to your TDD: If it isn't causing a problem then don't fix it. Don't over design.
In practice I find that if the static methods are tested well first then they are not going to be the cause of bugs in their callers.
If the static methods execute quickly then they don't need a mock.
If the static methods work with stuff from outside the program, then you might need a mock method. In this case you'd need to be able to simulate many different kinds of function behavior.
If you do need to mock a static method remember that there are ways to do it outside of OO programming.
For example, you can write scripts to process your source code into a test form that calls your mock function. You could link different object files that have different versions of the function into the test programs. You could use linker tricks to override the function definition (if it didn't get inlined). I am sure there are some more tricks I haven't listed here.
It's easy to test the static method. The problem is that there is no way to isolate your other code from that static method when testing the other code. The calling code is tightly-coupled to the static code.
A reference to a static method cannot be mocked by many mocking frameworks nor can it be overridden.
If you have a class that is making lots of static calls, then to test it you have to configure the global state of the application for all of those static calls - so maintenance becomes a nightmare. And if your test fails, then you don't know which bit of code caused the failure.
Getting this wrong, is one of the reasons that many developers think TDD is nonsense. They put in a huge maintenance effort for test results that only vaguely indicate what went wrong. If they'd only reduced the coupling between their units of code, then maintenance would be trivial and the test results specific.
That advice is true for the most part.. but not always. My comments are not C++ specific..
writing tests for static methods (which are pure/stateless functions): i.e. the work off the inputs to produce a consistent result. e.g. Add below - will always give the same value given a particular set of inputs. There is no problem in writing tests for these or code that calls such pure static methods.
writing tests for static methods which consume static state : e.g. GetAddCount() below. Calling it in multiple tests can yield different values. Hence one test can potentially harm the execution of another test - tests need to be independent. So now we need to introduce a method to reset static state such that each test can start from a clean slate (e.g. something like ResetCount()).
Writing tests for code that accesses static methods but no source-code access to the dependency: Once again depends on the properties of the static methods themselves. However if they are gnarly, you have a difficult dependency. If the dependency is an object, then you could add a setter to the dependent type and set/inject a fake object for your tests. When the dependency is static, you may need a sizable refactoring before you can get tests running reliably. (e.g. Add an object middle-man dependency that delegates to the static method. Now plugin a fake middle-man for your tests)
Lets take an example
public class MyStaticClass
{
static int __count = 0;
public static int GetAddCount()
{ return ++__count; }
public static int Add(int operand1, int operand2)
{ return operand1 + operand2; }
// needed for testability
internal static void ResetCount()
{
__count = 0;
}
}
...
//test1
MyStaticClass.Add(2,3); // => 5
MyStaticClass.GetAddCount(); // => 1
// test2
MyStaticClass.Add(2,3); // => 5
//MyStaticClass.ResetCount(); // needed for tests
MyStaticClass.GetAddCount(); // => unless Reset is done, it can differ from 1