I created a test case with Catch2 and I am trying to use TrompeLoeil for finer testing.
However the mock calls either don't happen at all, or happen but don't seem to be detecting by the framework.
Here is a minimal reproductible example :
class MyRealClass
{
public:
virtual int returnSumOfValues(int a, int b)
{
return a + b;
}
};
class MyMockClass : public MyRealClass
{
public:
MAKE_MOCK2(returnSumOfValues, int(int,int));
};
class MyTestedClass
{
public:
MyTestedClass(MyRealClass* _computeObj)
: computeObj(_computeObj)
{
}
int compute()
{
return computeObj->returnSumOfValues(2,3);
}
MyRealClass* computeObj;
};
TEST_CASE("Testing Trompe L'oeil")
{
auto mock = new MyMockClass();
MyTestedClass testedClass(mock);
int val = testedClass.compute();
CHECK(val == 5);
REQUIRE_CALL(*mock, returnSumOfValues(2,3)).RETURN(5);
}
And here is the error I get from running this test :
------------------------------------------------------------------------------- Testing Trompe L'oeil
------------------------------------------------------------------------------- ../../../src/MEGAAutoTests/UnitTests/control/TransferBatchTests.cpp:164
...............................................................................
../../../src/MEGAAutoTests/UnitTests/main.cpp:38: FAILED: explicitly
with message: No match for call of returnSumOfValues with signature
int(int,int) with.
param _1 == 2
param _2 == 3
I debugged this step by step and the mocked returnSumOfValues() is the one being executed.
However, if I make MyRealClass::returnSumOfValues() not virtual, the mock class is not used at all.
And the error is :
------------------------------------------------------------------------------- Testing Trompe L'oeil
------------------------------------------------------------------------------- ../../../src/MEGAAutoTests/UnitTests/control/TransferBatchTests.cpp:164
...............................................................................
../../../src/MEGAAutoTests/UnitTests/main.cpp:43: FAILED: CHECK(
failure.empty() ) with expansion: false with message: failure :=
"../../../src/MEGAAutoTests/UnitTests/control/TransferBatchTests.
cpp:172 Unfulfilled expectation: Expected
*mock.returnSumOfValues(2,3) to be called once, actually never called
param _1 == 2
param _2 == 3 "
This seems to not be consistent with the official documentation, which states that :
The line MAKE_MOCK2(log, void(int severity, const std::string& msg))
creates a mock function void Logger::log(int, const std::string&). If
MAKE_MOCKn(...) or MAKE_CONST_MOCKn(...) are used to implement a
virtual function from a base class, it is always recommended to add a
third macro parameter override since it gives the compiler an ability
to complain about mistakes.
Recommended, not required. And to give the compiler more information, not to make the test work.
TL, DR
Why is my code sample not working?
Why do I need to make mocked functions virtual when the documentation suggests it is not mandatory?
The issue was that expectations need to be set beforehand.
I though they worked like assertions, checking a state after the code was executed, but this is not the case.
In this case, moving the call solved the problem. Here is the fixed code sample :
TEST_CASE("Testing Trompe L'oeil")
{
auto mock = new MyMockClass();
MyTestedClass testedClass(mock);
REQUIRE_CALL(*mock, returnSumOfValues(2,3)).RETURN(5);
int val = testedClass.compute();
CHECK(val == 5);
}
This is the solution to the first problem, the second one (only virtual functions can be mocked) remains unanswered.
Related
Given those interfaces:
class ITemperature
{
public:
virtual ~ITemperature() = deafult;
virtual int get_temp() const = 0;
};
class IHumidity
{
public:
virtual ~IHumidity() = deafult;
virtual int get_humidity() const = 0;
};
And this SUT:
class SoftwareUnderTest
{
public:
SoftwareUnderTest(std::unique_ptr<ITemperature> p_temp,
std::unique_ptr<IHumidity> p_humidity)
: m_temp{std::move(p_temp)}, m_humidity{std::move(p_humidity)}
{}
bool checker()
{
assert(m_temp && "No temperature!");
if (m_temp->get_temp() < 50)
{
return true;
}
assert(m_humidity && "No humidity");
if (m_humidity->get_humidity() < 50)
{
return true;
}
return false;
}
private:
std::unique_ptr<ITemperature> m_temp;
std::unique_ptr<IHumidity> m_humidity;
};
And this mocks:
class MockITemperature : public ITemperature
{
public:
MOCK_METHOD(int, get_temp, (), (const override));
};
class MockIHumidity : public IHumidity
{
public:
MOCK_METHOD(int, get_humidity, (), (const override));
};
I want to make a test that checks that get_temp is called and also that the second assert (the one that checks that the humidity is nullptr), but when a do this test, I get the assert, but I the expectation tells me that it's never called (but it is actually called once)
this is the test:
class Fixture : pu`blic testing::Test
{
protected:
void SetUp() override
{
m_sut = std::make_unique<SoftwareUnderTest>(m_mock_temperature, m_mock_humidity);
}
std::unique_ptr<StrickMockOf<MockITemperature>> m_mock_temperature = std::make_shared<StrickMockOf<MockITemperature>>();
std::unique_ptr<StrickMockOf<MockIHumidity>> m_mock_humidity;
std::unique_ptr<SoftwareUnderTest> m_sut;
};
TEST_F(Fixture, GIVEN_AnInvalidHumidityInjection_THEN_TestMustDie)
{
EXPECT_CALL(*m_mock_temperature, get_temp).Times(1);
ASSERT_DEATH(m_sut->checker(), "No humidity");
}
Apparently, this is a known limitation, see here and here.
From what I have managed to discover by experimentation so far:
If you can live with the error message about leaking mocks (haven't checked if it's true or a false positive, suppressing it by AllowLeak triggers the actual crash), it can be done by making the mocks outlive the test suite and then wrapping references/pointers to them in one more interface implementation.
//mocks and SUT as they were
namespace
{
std::unique_ptr<testing::StrictMock<MockIHumidity>> mock_humidity;
std::unique_ptr<testing::StrictMock<MockITemperature>> mock_temperature;
}
struct MockITemperatureWrapper : MockITemperature
{
MockITemperatureWrapper(MockITemperature* ptr_) : ptr{ptr_} {assert(ptr);}
int get_temp() const override { return ptr->get_temp(); }
MockITemperature* ptr;
};
struct Fixture : testing::Test
{
void SetUp() override
{
mock_temperature
= std::make_unique<testing::StrictMock<MockITemperature>>();
m_mock_temperature = mock_temperature.get();
// testing::Mock::AllowLeak(m_mock_temperature);
m_sut = std::make_unique<SoftwareUnderTest>(
std::make_unique<MockITemperatureWrapper>(m_mock_temperature), nullptr);
}
testing::StrictMock<MockITemperature>* m_mock_temperature;
std::unique_ptr<SoftwareUnderTest> m_sut;
};
TEST_F(Fixture, GIVEN_AnInvalidHumidityInjection_THEN_TestMustDie)
{
EXPECT_CALL(*m_mock_temperature, get_temp).WillOnce(testing::Return(60));
ASSERT_DEATH(m_sut->checker(), "No humidity");
}
https://godbolt.org/z/vKnP7TsrW
Another option would be passing a lambda containing the whole to ASSERT_DEATH:
TEST_F(Fixture, GIVEN_AnInvalidHumidityInjection_THEN_TestMustDie)
{
ASSERT_DEATH(
[this] {
EXPECT_CALL(*m_mock_temperature, get_temp)
.WillOnce(testing::Return(60));
m_sut->checker();
}(), "No humidity");
}
Works, but looks ugly, see here.
Last but not least: one can use custom assert or replace__assert_failed function and throw from it (possibly some custom exception), then use ASSERT_THROW instead of ASSERT_DEATH. While I'm not sure replacing __assert_failed is legal standard-wise (probably not), it works in practice:
struct AssertFailed : std::runtime_error
{
using runtime_error::runtime_error;
};
void __assert_fail(
const char* expr,
const char *filename,
unsigned int line,
const char *assert_func )
{
std::stringstream conv;
conv << expr << ' ' << filename << ' ' << line << ' ' << assert_func;
throw AssertFailed(conv.str());
}
Example: https://godbolt.org/z/Tszv6Echj
I want to make a test that checks that get_temp is called and also
that the second assert (the one that checks that the humidity is
nullptr), but when a do this test, I get the assert, but I the
expectation tells me that it's never called (but it is actually called
once)
First you have to understand how death test are working.
Before executing code in a macro ASSERT_DEATH gtest forks test process so when death happens test can continue.
Then forked process is executing code which should lead to process death.
Now test process joins forked process to see result.
Outcome is that in one process checker() is executed and mock is invoked and in test process it is checker() is not invoked so also mock is not invoked. That is why you get an error that mock is not satisfied.
Now answer from alager makes mock eternal so missing expected call is not reported. And since code uses global state adding other tests will lead to some problems. So I would not recommend this approach.
After edit he moved EXPECT_CALL inside ASSERT_DEATH so now only forked process expects call, but this is not verified since process dies before mock is verified. So again I would not recommend this approach either.
So question is what you should do? IMO your problem is that you are testing to much of implementation details. You should loosen test requirement (drop StrictMock or make it even make it NiceMock). Still I find this a bit clunky. Live demo
I would change code in such way that it is impossible to construct SoftwareUnderTest with nullptr dependencies. You can use gsl::not_null for that.
It seems to be due to some tricky mechanism that is used in googletest to assert death (they mention creating a child process). I did not find a way to fix it correctly, but I found one (not so great) workaround:
SoftwareUnderTest(ITemperature* p_temp, IHumidity* p_humidity) // changed signature to allow leaks, I guess you cannot really do it in the production
and then:
class Fixture : public testing::Test
{
public:
Fixture(): m_mock_temperature(new MockITemperature), m_mock_humidity(nullptr) {}
~Fixture() {
// Mock::VerifyAndClearExpectations(m_mock_temperature); // if I uncomment that, for some reason the test will fail anyway
std::cout << "Dtor" << std::endl;
// delete m_mock_temperature; // if I delete the mock correctly, the test will fail
}
protected:
void SetUp() override
{
// m_sut.reset(new SoftwareUnderTest(m_mock_temperature.get(), m_mock_humidity.get()));
m_sut.reset(new SoftwareUnderTest(m_mock_temperature, m_mock_humidity));
}
// std::unique_ptr<MockITemperature> m_mock_temperature; // if I use smart pointers, the test will fail
// std::unique_ptr<MockIHumidity> m_mock_humidity;
MockITemperature* m_mock_temperature;
MockIHumidity* m_mock_humidity;
std::unique_ptr<SoftwareUnderTest> m_sut;
};
TEST_F(Fixture, GIVEN_AnInvalidHumidityInjection_THEN_TestMustDie)
{
EXPECT_CALL(*m_mock_temperature, get_temp).Times(1).WillOnce(Return(60)); // this is to ensure to go over first check, seems you forgot
ASSERT_DEATH(m_sut->checker(), "No humidity");
std::cout << "after checks" << std::endl;
}
Sorry, that's all I could figure out for the moment. Maybe you can submit a new issue in gtest github while waiting for a better answer.
I am a beginner with google testing framework and have looked up for the solution to this question on SO, but could not find any solutions with respect to C++. Anyway here is what i am trying to do. I have a state machine(service) which is called inside a client code.
//IStateMachine.h
class IStateMachine
{
public:
bool Run(const std::string& action) = 0;
bool IsTxnValid(const std::string& action)= 0;
}
//StateMachine.h
class StateMachine : public IStateMachine
{
bool Run(const std::string& action) override;
bool IsTxnValid(const std::string& action) override;
}
//StateMachine.cpp
bool StateMachine::IsTxnValid(const std::string& action)
{
//Checks whether the given action is valid for the given state.
}
bool StateMachine::Run(const std::string& action)
{
if(IsTxnValid(action)) // #E
{
//Do processing
return true;
}
return false;
}
//Client.h contains a class Client which has function called RunService.
Client
{
public:
void RunService();
std::unique_ptr<IStateMachine> service_; // Initialised to a non null value in either ctr or
// factory.
}
//Client.cpp
bool Client::RunService(std::string&action)
{
if(!service_->Run(action)) //Run in turn calls IsTxnValid().
{
return false;
}
return true;
}
Now i am writing a test case to test the functioning of RunService. I am expecting that if Client::IsTxnValid(param) returns false, then so should RunService.
I have successfully set up the testing recipe and could get the basic tests running. Here is the relevant test i have written. On running this test the i get the error, that IsTransitionValid is never called.
TEST_F(ClientTest, RunService)
{
EXPECT_CALL(*p_service, Run("some_action")); // #A
// EXPECT_CALL(*p_service, Run(testing::_)).WillOnce(::testing::Return(true)); //#B
EXPECT_CALL(*p_service,IsTransitionValid(testing::_)).WillOnce(::testing::Return(false)); //#C : This never gets called.
EXPECT_EQ(false, x_client->RunService());
}
How do i correctly call IsTransitionValid ?
You don't need to set this expectation. I'd go even further: you should not even depend on the implementation of Run in IStateMachine: you should only care about what input it is provided with (parameters, checked with matchers) and what output it can return (so basically only the contract between these two classes) and that's the beauty of it!
It is an implementation detail of StateMachine class (the real implementation) what is done when Run is called. The only thing you need to check in your test is to act upon the result of Run. Using triple A rule (arrange, act, assert): you arrange the test case conditions (using EXPECT_CALLs), then you act (calling RunService) and then you assert (checking the result of RunService).
The technical details:
When you create a mock by inheriting from class Foo:
class Foo {
public:
virtual ~Foo() = default;
virtual void bar() = 0;
}
By defining:
class FooMock : public Foo {
MOCK_METHOD0( bar, void());
}
gmock will add bar (the method to override) and gmock_bar (internal detail of gmock) methods to FooMock class. bar has empty implementation in this case. FooImpl and FooMock share the interface, but have different implementations - hence no call to IsTxnValid is made in Run: the mock class just doesn't know (nor care) how Run is implemented in StateMachine. Remember: in your testcase you interact with StateMachineMock and you only care about the interaction with its public interface, the contract between these two classes and how they cooperate together.
That being said, you of course need to utest the StateMachine class. It may depend on yet another interfaces in its implementations: that will be tested with different set of mocks. But Client should not know about this.
In this example bellow I will obtain the following behavior:
EXPECT_CALL(barMock, doBar(7))...
Expected arg #0: is equal to 7
Actual: 5
Expected: to be called once
Actual: called once - saturated and active
#include <gmock/gmock.h>
#include <gtest/gtest.h>
class IBar
{
public:
virtual bool doBar(int barParam) = 0;
};
class BarMock : public IBar
{
public:
MOCK_METHOD1(doBar, bool(int));
};
class Foo
{
public:
Foo(IBar& bar_)
: bar{ &bar_ }
{
}
void doFoo()
{
bar->doBar(7);
bar->doBar(5);
}
IBar* bar;
};
class FooBarTest : public ::testing::Test
{
public:
void SetUp() override
{
ON_CALL(barMock, doBar(testing::_)).WillByDefault(testing::Return(true));
}
testing::NiceMock<BarMock> barMock;
};
TEST_F(FooBarTest, OnCallExpectCallSameMethod)
{
Foo foo(barMock);
ON_CALL(barMock, doBar(5)).WillByDefault(testing::Return(true));
EXPECT_CALL(barMock, doBar(7)).WillOnce(testing::Return(true));
foo.doFoo();
}
Google test versions tested:
1.8.0
1.8.1
Is this intended to cause an error?
How does google test sets the
order calls? Does EXPECT_CALL has higher priority only in this
scenario?
Yes, it is intended to be an error. One of the GoogleMock policies is that in case of possible ambiguity it's better to throw an error an make user state his intention explicitly.
Adding EXPECT_CALL macro effectively says "I care about calls of this method". Whenever EXPECT_CALL is set, GoogleMock will try to match every EXPECT_CALL it has seen in reverse order of declaration (so the latest defined expects are matched first).
One of the reasons for this design is to allow overriding less specific expects with more specific ones (e.g. in constructor of test fixture you set less restrictive expects, but for certain tests you want to match more precisely). Documentation.
However, there is a way of "ignoring" already fulfilled expectations by adding .RetiresOnSaturation()
TEST_F(FooBarTest, OnCallExpectCallSameMethod)
{
Foo foo(barMock);
ON_CALL(barMock, doBar(5)).WillByDefault(testing::Return(true));
EXPECT_CALL(barMock, doBar(7))
.WillOnce(testing::Return(true))
.RetiresOnSaturation();
foo.doFoo();
}
RetiresOnSaturation() will make expectations retired after it has been saturated (i.e. called as many time as expected). GoogleMock will skip over retired expectations as long as there are still any non-retired ones (if all of them are retired then it still prints an error).
If you however need to accept calls doBar(5) before doBar(7), then the only way to do so is to define it as expectation as well:
TEST_F(FooBarTest, OnCallExpectCallSameMethod)
{
Foo foo(barMock);
EXPECT_CALL(barMock, doBar(5)).WillRepeatedly(testing::Return(true));
EXPECT_CALL(barMock, doBar(7))
.WillOnce(testing::Return(true))
.RetiresOnSaturation();
foo.doFoo();
}
RetiresOnSaturation() is still needed because of LIFO (last in, first out) processing of expectations.
I would like to test the method of my system, whose return value partially depends on the return value of the call to some kind of connection interface. In most cases I would like the IConnection to return true upon any kind of call to it's open(_, _) method. Except in one case, when I explicitly test for the condition with failed connection.
Example:
/*
* Some kind of network interface with method `open`
*/
class IConnection {
public:
IConnection() = default;
virtual ~IConnection() = default;
virtual bool open(const std::string& address, int port) = 0;
};
class ConnectionMock: public IConnection {
public:
MOCK_METHOD2(open, bool(const std::string& address, int port));
};
class MySystem {
public:
MySystem() = delete;
MySystem(std::shared_ptr<IConnection> connection): connection_(connection) {}
bool doSth() {
/*
* Do some things, but fail if connection fails
*/
bool connectionStatus = connection_->open("127.0.0.1", 6969);
if (!connectionStatus) {
return false;
}
// do other things
return true;
}
private:
std::shared_ptr<IConnection> connection_;
};
TEST(MySystemShould, returnFalseIfFailedToOpenConnectionAndTrueIfSucceeded) {
auto connectionMock = std::make_shared<NiceMock<ConnectionMock> >();
ON_CALL(*connectionMock, open(_, _)).WillByDefault(Return(true));
MySystem system(connectionMock);
// if I don't specify Times test fill fail, because WillOnce automatically sets Times(1)
EXPECT_CALL(*connectionMock, open(_, _)).Times(AnyNumber()).WillOnce(Return(false));
/*
* Commented code below is not a good solution - after expectation retires
* the test will fail upon subsequent calls
*/
//EXPECT_CALL(*connectionMock, open(_, _)).WillOnce(Return(false)).RetiresOnSaturation();
ASSERT_FALSE(system.doSth());
/*
* Code bellow allows me to avoid the warning
*/
//EXPECT_CALL(*connectionMock, open(_, _)).WillRepeatedly(Return(true));
ASSERT_TRUE(system.doSth());
}
The problems with my current solution is that when the EXPECT_CALL override becomes saturated, even though gmock goes back to the default action specified on ON_CALL, every subsequent call to open(_, _) is causing the following warning:
GMOCK WARNING:
/whatever.cpp:105: Actions ran out in EXPECT_CALL(*connectionMock, open(_, _))...
Called 2 times, but only 1 WillOnce() is specified - taking default action specified at:
/whatever.cpp:103:
even though I'm using NiceMock. I can get rid of the warning by specifying EXPECT_CALL with WillRepeatedly(Return(true)), but this is the duplication of my code in ON_CALL.
I would like to know, how can I override the default action specified with ON_CALL for just one call to IConnection::open, and then go back to the defaults, without causing gmock to print a warning. The perfect solution would be something similar to:
EXPECT_CALL(*connectionMock, open(_, _)).WillOnce(Return(false)).DisableExpectationAfterSaturation();
but it doesn't exist. RetiresOnSaturation doesn't work as I would like, because it fails the test after getting saturated (doesn't match action specified with ON_CALL).
EDIT 2
The DoDefault() - feature comes close to what is asked in the question. It specifies that an action in EXPECT_CALL should go back to the default action specified by ON_CALL:
using ::testing::DoDefault;
// Default action
ON_CALL(*connectionMock, open(_, _)).WillByDefault(Return(true));
// returns true once and then goes back to the default action
EXPECT_CALL(*connectionMock, open(_, _)
.WillOnce(Return(false))
.WillRepeatedly(DoDefault());
Initial answer
If the return value of IConnection::open depends on the parameters you can specify ON_CALL twice but with different arguments (or rather arguments instead of the placeholder):
ON_CALL(*connectionMock, open(_, _)).WillByDefault(Return(true));
ON_CALL(*connectionMock, open("BAD_ADDRESS", 20)).WillByDefault(Return(false));
So any time the mocked method open will be called with arguments "BAD_ADDRESS" and 20, it will return false, and true otherwise.
Here is a simple example:
using ::testing::_;
using ::testing::AnyNumber;
using ::testing::Return;
class A {
public:
virtual bool bla(int a) = 0;
};
class MOCKA : public A {
public:
MOCK_METHOD1(bla, bool(int));
};
TEST(ABC, aBABA) {
MOCKA a;
ON_CALL(a, bla(_)).WillByDefault(Return(false));
ON_CALL(a, bla(1)).WillByDefault(Return(true));
EXPECT_CALL(a, bla(_)).Times(AnyNumber());
EXPECT_TRUE(a.bla(1));
EXPECT_TRUE(a.bla(1));
EXPECT_TRUE(a.bla(1));
EXPECT_FALSE(a.bla(2));
EXPECT_FALSE(a.bla(3));
EXPECT_FALSE(a.bla(4));
}
EDIT 1
I think now I understood the problem and if I did then the solution is very simple:
EXPECT_CALL(*connectionMock, open(_, _))
.Times(AnyNumber())
.WillOnce(Return(true))
.WillRepeatedly(Return(false));
When ConnectionMock::open will be called inside of MySystem::doSth it will once return true and then always return false no matter what the arguments are. In this case you also don't need to specify ON_CALL. Or do you definitely need to specify the actions with ON_CALL instead of EXPECT_CALL?
I am trying to test a class similar to the example below:
public class Service : IService
{
public string A(string input)
{
int attemptCount = 5;
while (attemptCount > 0)
{
try
{
return TryA(input);
}
catch (ArgumentOutOfRangeException)
{
attemptCount--;
if (attemptCount == 0)
{
throw;
}
// Attempt 5 more times
Thread.Sleep(1000);
}
}
throw new ArgumentOutOfRangeException();
}
public string TryA(string input)
{
// try actions, if fail will throw ArgumentOutOfRangeException
}
}
[TestMethod]
public void Makes_5_Attempts()
{
// Arrange
var _service = MockRepository.GeneratePartialMock<Service>();
_service.Expect(x=>x.TryA(Arg<string>.Is.Anything)).IgnoreArguments().Throw(new ArgumentOutOfRangeException());
// Act
Action act = () => _service.A("");
// Assert
// Assert TryA is attempted to be called 5 times
_service.AssertWasCalled(x => x.TryA(Arg<string>.Is.Anything), opt => opt.Repeat.Times(5));
// Assert the Exception is eventually thrown
act.ShouldThrow<ArgumentOutOfRangeException>();
}
The partial mocking doesn't seem to accept my expectation. When I run the test I receive an error about the input. When I debug, I see that the actual implementation of the method is being executed instead of the expectation.
Am I doing this test correctly? According to the documentation ( http://ayende.com/wiki/Rhino%20Mocks%20Partial%20Mocks.ashx ): "A partial mock will call the method defined on the class unless you define an expectation for that method. If you have defined an expectation, it will use the normal rules for this."
It's is important to note that mocking frameworks like Rhinomocks, Moq and NSubstitute use a feature in .NET called DynamicProxy that dynamically generates a derived class of the mock in memory. Classes must:
be an interface; or
non-sealed class with parameterless constructor; or
derive from MarshalByRefObject (moq has moved away from this feature)
Methods must be part of the interface or made virtual so that alternate behaviors can be substituted at runtime.