Since the datastructure of my application domain is becoming pretty complex as of late, I started reading up on mock objects. Soon a simple question came to my mind, but the answer has proven to be quite the headache so far. So here goes:
We have a class 'Foo' with 'bar' as one of its methods:
class Foo {
public String bar(int i){
if(i == 1) return "arrr!";
}
}
And we have a class Pirate calling Foo.bar(1); in one of its methods:
class Pirate {
public String yell(){
Foo foo = new Foo();
return foo.bar(1);
}
Now we mock the Foo class in the unit test of the Pirate class because Foo happens to have a plethora of other dependencies:
#Test
public void returnsPirateString() {
Pirate blackBeard = new Pirate();
Foo fooMock = mock(Foo.class);
fooMock.expectAndReturn("bar",1,"arrr!"); //expects 'bar' function to be called once and returns "arrr!"
assertEquals(blackBeard.yell(),"arrr!");
}
What happens now, is that if we refactor method bar to return null instead of "arrr!", our test will keep running happily while our program does not work the way we want it to. This may result in a possible debugging nightmare.
Using a mockist approach instead of the classical testing approach to unit testing, most of the time all "helper" objects get mocked, and only the tested object remains unmocked, so the previously stated problem can occur quite often as well.
What can be done to prevent this problem while mocking?
In your test, you are testing the yell() method of Pirate class which uses Foo. So you have to mock the behavior of Foo's bar method.
To make sure your bar method is functioning correctly, you need another test case to test the bar method of Foo.
#Test
public void testBar() {
//make sure bar retrun "arrr"!
}
Now if your bar method returns null, this test case will fail!
You should be testing the 'helper object' in isolation as well. Once both of those are covered and tested, then you can be sure that both interact with each other in the expected way.
Changing your 'helper object' is something that should be done with tests against that helper object to confirm it still behaves as expected.
If you are concerned about the specific runtime behavior of the combination of helper and primary class, then you should use integration tests, or some other test at a higher level, to assert the two work together as expected.
The test returnsPirateString is not a false positive - it's testing what happens when a Pirate's Foo instance returns 'arrr!'
In other words, when you're testing Pirate.yell, it doesn't matter what Foo.bar returns, unless it creates a special boundary condition (and you should probably already have a test that documents what yell does when Foo returns null).
Pirate.yell is not responsible for guaranteeing any particular return value for Foo.bar, so its unit tests should not expect any particular return values.You should even make a point of changing your test to use something other than the current return value of Foo.bar.
Related
I have this problem. I have a method in a class which gives me some return value. This method also makes changes to the state of the instance of that class. I need to unit test them both, so I made the test cases for the return value and copied them and just changed the assert from testing the return value to testing the change in representation (the only way for me to test this class is to test its representation). However, this does not seem like a very good way to deal with this. If I ever have changes in the functionality of the class, I will have to change the Act in the AAA in both test cases. How do I go about testing this? Is there some pattern to do that? Having both asserts in one test would mean I would be testing two things in one test case. In some languages (like JavaScript), I know I can make an array of tuples with the functionality, the assert for the return value and the assert for the representation, but I'm not sure a) how much better that is over some copy-pasting and b) what to do in other languages (e.g. in C# I imagine I'd have to make some classes with the asserts and include them in test cases).
Edit: For example:
class A {
state;
foo(animal, foodAmount) {
let returnResult = {};
//does things with animal and foodAmount
//which things change BOTH returnResult and the state variable
return returnResult;
}
bar() {
let stateToReturn = "";
//state is used to change the stateToReturn variable
return stateToReturn;
}
}
Here I will test function foo with a) an animal that does not exist currently, b) an animal that exists where I will add a non-zero amount of food to make sure the amount of food changes, c) two different animals to make sure they do not interfere with each other's food amounts, etc. I will do all these tests and make sure the returnResult variable is correct. The problem is that I need to make sure it does not only affect the returnResult variable, but also the state. The bar function is the only way for me to see the inner state. How would I go about testing the foo function? Would I write all tests twice (once to check foo returns the correct result and once to check foo changes the state correctly)?
Even with your example it's still a bit vague. The general and short answer to your question is: You should test for behavior, not methods. If that requires you to make assertions on the result of multiple methods in one test that's perfectly fine.
You shouldn't, however, expose the internal state of your class just to make it testable. It's better to test against the public api of your class. After all, that's how the production code interacts with it.
If a method changes the internal state but there is nothing in the public api that makes the change visible, how do you test it? You probably have a hidden concept in your class that should be extracted into its own class with its own public api. You can test the new class independently and the interaction between the two classes. This leads to smaller and more focused tests.
I'm not sure if this answers your question sufficiently but maybe there's some food for thought.
If i need to test if a method within class under test has been called or not, can it be done without Mockito (or any mocking tool for that matter)?
Reason asking is that wherever i read about Mockito and similar tools, it says one should never mock CUT but its dependencies (that part is clear).
So, if thats the case then there are only 2 options left:
there is some other way of testing it without mocking
or
the fact the method was called should not be tested itself but some side effect or methods return value
For example (trivial and non-realworld), class MyClass can have 2 methods: A() and B(). A conditionay calls B based on some internal state.
After arranging state & acting by calling A() we want to assert that B() was called.
Either its not possible without mocking the whole CUT or 2 methods like this in a single class are always SRP violation smell and call for redesign where B() should actually be (mocked) dependency of MyClass CUT.
So, whats correct?
Usually I tend to not even use spies, instead I prefer to write my code in a way that for any class I write:
I test only non-private methods, since they're entry points into the class under test. So, in your example, if a() calls b(), maybe b() should be be private and, as a consequence, should not be tested. To generalize, a() is something that a class "can do" (a behavior), so I test the behavior, and not the method itself. If this behavior internally calls other things - well, its an internal matter of that class, if possible I don't make any assumptions on how does the class work internally, and always prefer "white-box" testing.
I only test "one" non-private method in a test.
All the methods should return something (best option) or at least call dependencies, or change internal state of the object under test. The list of dependencies is always clean-to-understand, I can't instantiate the object of CUT without supplying it a list of dependencies. For example, using constructor dependency injection is a good way of doing this. I mock only dependencies indeed, and never mock / spy CUT. Dependencies are never static but injected.
Now with these simple rules, the need to "test if a method within class under test has been called or not" basically can boil down to one of the following:
you're talking about private method. In this case - don't test it, test only public things.
The method is public - in this case you explicitly call it in unit test, so its irrelevant.
Now lets ask why do you want to test this if a method within CUT has been called or not?
If you want to make sure that it changed something. If this "something" is within the class - in other words, its internal state has changed, check in test that the change is indeed done in the state by calling another method that allows to query the state
If this "something" is a code that is managed by dependency, create a mock of this dependency and verify that it was called with the expected parameters.
Take a look at the Mockito Documentation (https://static.javadoc.io/org.mockito/mockito-core/3.0.0/org/mockito/Mockito.html#13)
When using a Spy you can 'replace' a method in the same class that is under test.
#ExtendWith(MockitoExtension.class)
public class Test {
class MyClass {
public void a() {
b();
}
public void b() {
}
}
#Test
public void test() {
MyClass testClass = new MyClass();
MyClass spy = Mockito.spy(testClass);
Mockito.doNothing().when(spy).b();
spy.a();
Mockito.verify(spy, Mockito.times(1)).b();
}
}
So whether that is something that should be done is a different question ;)
I think it highly depends on what method B() is actually doing and whether that is supposed be part of MyClass in the first place.
Either its not possible without mocking the whole CUT
In this case we do not mock the whole CUT only the method you do not want to be called.
Reason asking is that wherever i read about Mockito and similar tools, it says one should never mock CUT but its dependencies (that part is clear).
I believe this statement is not entirely accurate in correlation with spying.
The whole point of spying in my eyes is to use it on the class under test. Why would one want to spy on a dependecy that is not even supposed to be part of the test in the first place?
I am a long time lurker in the board, and is needless to say that you guys are the best and I am grateful for all the times you saved my job. That is my first time post here and I hope I don't mess it up.
I am writing a C++ Boost application for Linux (Virtualized Ubuntu 16.04 amd_64) and am using Turtle Mock for the mocking framework and Boost Test for the testing framework. When I try to test a class which uses dependency injection technique, I mock the classes that need to be given to the testing class so I can verify the sequence of their invocation. S far so good, but the problem comes here.
I am using MOCK_BASE_CLASS(MockAClass, AClass), to override the virtual methods of the real AClass, and use to new MockAClass to proceed with my tests. Let's say AClass has a virtual method int getTest(int), and MockAClass has MOCK_METHOD(getTest, 1, int(int)), after setting expectation and return value for the getTest method of MockAClass object, and invoking the method, the expectation which in most cases is MOCK_EXPECT(objMockAClass.getTest).at_least(1) is NEVER verified. I can control the return value, but the call is never verified as it happened. This only occurs if the function returns a value (for ex. if the function is void getTest(int) then the verification will pass).
I am attaching a simple PoC of my problem that will fail on my system.
class AClass
{
public:
virtual int getTest(int a) {return 0}
}
MOCK_BASE_CLASS (MockAClass, AClass)
{
MOCK_METHOD(getTest, 1, int(int));
}
BOOST_AUTО_TEST_CASE(SomeClassFunctionality)
{
MockAClass objMockAClass;
MOCK_EXPECT(objMockAClass.getTest).returns(1);
MOCK_EXPECT(objMockAClass.getTest).at_least(1);
objMockAClass.getTest(1);
}
MOCK_EXPECT(objMockAClass.getTest).returns(1);
MOCK_EXPECT(objMockAClass.getTest).at_least(1);
This is actually two expectations. The first one means 'everytime getTest gets called return 1' and the second 'getTest must be called at least one time'.
The problem is that the first one will always match, therefore the second will not have a chance to be triggered.
Тhe problem is solved if separate EXPECT statements are combined in one whole EXPECT statement.
MOCK_EXPECT(objMockAClass.getTest).at_least(1).returns(1); - This will make the example work as planned.
Regards,
I often face the problem that mock objects need to be brought in a certain state before the "interesting" part of a test can start.
For example, let's say I want to test the following class:
struct ToTest
{
virtual void onEnable();
virtual void doAction();
};
Therefore, I create the following mock class:
struct Mock : ToTest
{
MOCK_METHOD0(onEnable, void());
MOCK_METHOD0(doAction, void());
};
The first test is that onEnable is called when the system that uses a ToTest object is enabled:
TEST(SomeTest, OnEnable)
{
Mock mock;
// register mock somehow
// interesting part of the test
EXPECT_CALL(mock, onEnable());
EnableSystem();
}
So far, so good. The second test is that doAction is called when the system performs an action and is enabled. Therefore, the system should be enabled before the interesting part of the test can start:
TEST(SomeTest, DoActionWhenEnabled)
{
Mock mock;
// register mock somehow
// initialize system
EnableSystem();
// interesting part of the test
EXPECT_CALL(mock, doAction());
DoSomeAction();
}
This works but gives an annoying warning about an uninteresting call to onEnable. There seem to be two common fixes of this problem:
Using NiceMock<Mock> to suppress all such warnings; and
Add an EXPECT_CALL(mock, onEnable()) statement.
I don't want to use the first method since there might be other uninteresting calls that really should not happen. I also don't like the second method since I already tested (in the first test) that onEnable is called when the system is enabled; hence, I don't want to repeat that expectation in all tests that work on enabled systems.
What I would like to be able to do is say that all mock calls up to a certain point should be completely ignored. In this example, I want expectations to be only checked starting from the "interesting part of the test" comment.
Is there a way to accomplish this using Google Mock?
The annoying thing is that the necessary functions are there: gmock/gmock-spec-builders.h defines Mock::AllowUninterestingCalls and others to control the generation of warnings for a specific mock object. Using these functions, it should be possible to temporarily disable warnings about uninteresting calls.
That catch is, however, that these functions are private. The good thing is that class Mock has some template friends (e.g., NiceMock) that can be abused. So I created the following workaround:
namespace testing
{
// HACK: NiceMock<> is a friend of Mock so we specialize it here to a type that
// is never used to be able to temporarily make a mock nice. If this feature
// would just be supported, we wouldn't need this hack...
template<>
struct NiceMock<void>
{
static void allow(const void* mock)
{
Mock::AllowUninterestingCalls(mock);
}
static void warn(const void* mock)
{
Mock::WarnUninterestingCalls(mock);
}
static void fail(const void* mock)
{
Mock::FailUninterestingCalls(mock);
}
};
typedef NiceMock<void> UninterestingCalls;
}
This lets me access the private functions through the UninterestingCalls typedef.
The flexibility you're looking for is not possible in gmock, by design. From the gmock Cookbook (emphasis mine):
[...] you should be very cautious about when to use naggy or strict mocks, as they tend to make tests more brittle and harder to maintain. When you refactor your code without changing its externally visible behavior, ideally you should't need to update any tests. If your code interacts with a naggy mock, however, you may start to get spammed with warnings as the result of your change. Worse, if your code interacts with a strict mock, your tests may start to fail and you'll be forced to fix them. Our general recommendation is to use nice mocks (not yet the default) most of the time, use naggy mocks (the current default) when developing or debugging tests, and use strict mocks only as the last resort.
Unfortunately, this is an issue that we, and many other developers, have encountered. In his book, Modern C++ Programming with Test-Driven Development, Jeff Langr writes (Chapter 5, on Test Doubles):
What about the test design? We split one test into two when we changed from a hand-rolled mock solution to one using Google Mock. If we expressed everything in a single test, that one test could set up the expectations to cover all three significant events. That’s an easy fix, but we’d end up with a cluttered test.
[...]
By using NiceMock, we take on a small risk. If the code later somehow changes to invoke another method on the [...] interface, our tests aren’t going to know about it. You should use NiceMock when you need it, not habitually. Seek to fix your design if you seem to require it often.
You might be better off using a different mock class for your second test.
class MockOnAction : public ToTest {
// This is a non-mocked function that does nothing
virtual void onEnable() {}
// Mocked function
MOCK_METHOD0(doAction, void());
}
In order for this test to work, you can have onEnable do nothing (as shown above). Or it can do something special like calling the base class or doing some other logic.
virtual void onEnable() {
// You could call the base class version of this function
ToTest::onEnable();
// or hardcode some other logic
// isEnabled = true;
}
I have a quite complicated method for which I want to test the behavior (using Mockito and JUnit). This method takes an object (let's call its type State) as input, and should take a few different state variables into account for deciding its output.
As an example, considering the following specification (s is a mock of the State class):
If s.varOne is set, return its value.
Else, if s.varTwo is set, return that instead.
Else, if s.varThree is set, call s.update(s.varThree) and then return s.varOne, which will now have a value (even though it didn't at stage 1.)
Else, throw an error.
In order to test case 3 properly, I would like to set up the s object so that s.varOne and s.varTwo are both unset to begin with, but if (and only if!) the sut calls s.update(s.varThree), then after that s.varOne returns something.
Is there a good way to setup this behavior in Mockito?
I have considered setting up some chain of return values for s.varOne, and then verifying that the order of the calls corresponds to the order of the outputs (as well as that the return value of the sut is correct, of course), but this feels dirty; if I then change the method to calculate its return value in some other way, which calls s.varOne a different number of times but doesn't change its output, then the test will fail even though the functionality is the same.
My ideal solution is a way where I can add some "delayed" setup for the mock object, which is run when the sut calls the s.update() method, but I can't figure out a way to accomplish that.
You have a couple of options to mock a state change here, a good option and a better option. The best option, as tieTYT notes above, is to just to untangle the general contract of State: Does it really make sense for State to be mutable, and to have self-mutating methods that aren't simple setters?
The good option (sometimes) is to create a disposable subclass—or, better, an interface implementation—of State.
#Test public void yourTest() {
SystemUnderTest sut = createSystemUnderTest();
State state = new State() {
#Override public void update() {
this.varOne = 42;
}
}
// rest of your test
}
At that point, you may not need Mockito at all. Be warned, though: This can get a little tricky if State has side-effects or is otherwise difficult to instantiate. You could extract an interface, which would require you to wrap your fields in getters; you could also make State a named abstract class and then pass mock(MyAbstractState.class, CALLS_REAL_METHODS), but that gets particularly hairy when you consider that no initializer actually runs on that fake instance, and consequently the fields are all zero or null. If it's not simple, don't waste your time forcing a square peg into a round hole.
A more-common pattern for mocks is to use a custom Answer, in which you can execute arbitrary code at the expense of type safety. Here's an example, assuming update is void and varThree is an integer:
#Test public void yourTest() {
SystemUnderTest sut = createSystemUnderTest();
final State s = Mockito.mock(State.class);
doAnswer(new Answer<Void>() {
#Override public Void answer(InvocationOnMock invocation) {
int actualVarThree = (int) invocation.getArguments()[0];
assertEquals(EXPECTED_VAR_THREE, actualVarThree);
s.varOne = actualVarThree;
return null;
}
}).when(s).update(anyInt());
// rest of your test
}
Note that the array of arguments is an Object[], so the cast is necessary and slightly-dangerous, but then you can make all sorts of assertions and modifications synchronously when your mocked method is called.
Hope that helps!