Suppose I have class Car with following methods:
LoadGasoline(IFuel gas)
InsertKey(IKey key)
StartEngine()
IDrivingSession Go()
the purpose of Car is to configure and return an IDrivingSession which the rest of the application uses to drive the car. How do I unit-test my Car?
It looks like it requires a sequence of operations done before I can call Go() method. But I want to test each method separately as they all have some important logic. I don't want to have bunch of unit-tests like
Test1: LoadGasoline, Assert
Test2: LoadGasoline, InsertKey, Assert
Test3: LoadGasoline, InsertKey, StartEngine, Assert
Test4: LoadGasoline, InsertKey, StartEngine, Go, Assert
Isn't there a better way to unit-test sequential logic or is this a problem with my Car design?
--- EDIT ----
Thanks for all the answers. As many noticed, I should also have tests for invalid scenarios and I have those too, but this question is focused on how to test the valid sequence.
I think each method should be tested separately and independently.
IMHO, you should prepare the environment for each case, so only the LoadGasoline test will break if you change the LoadGasoline method, and you won't need to see all the tests break because of a single bug.
I don't know how the state of your Car looks like, but, before the InsertKey, you should prepare with a method like, car.SetTotalGasoline(20); or whatever variable is set in this method, but not depend on a complex logic of the method LoadGasoline.
You will later need a test (in this case, not a unit test) to test all the sequence.
Some unit testing frameworks let you specify set-up code which runs before the actual test starts.
This allows you to get the target object into the proper state before running your test. That way your test can pass or fail based on the specific code you're testing rather than on the code needed before you can run a test.
As a result, your test sequence will wind up something like this:
Test1:
LoadGasoline, Assert
Test2 Setup:
LoadGasoline
Test2:
InsertKey, Assert
Test3 Setup:
LoadGasoline, InsertKey
Test3:
StartEngine, Assert
Test4 Setup:
LoadGasoline, InsertKey, StartEngine
Test4:
Go, Assert
Realistically speaking, since the tests are all run in sequence there's no chance of Test's Setup failing if the previous test passes.
With that said, you should also test failure cases that aren't expected to work but that's a different issue.
Why don't you want all those tests?
Go has very different behavior if you call it before or after, say, InsertKey, right? So you ought to be testing both behaviors, in my opinion.
Its a fair reluctance but sometimes thats what you need to do. If you can't fake out the system under test so it thinks its in a later state, then you need to go through the same steps to get it into that state. Without knowing more about what your testing its not clear how you could fake out the different states.
One way you can make this tolerable is use an extract method refactoring on tests for one state so that same code can be used to prepare the next test.
I would probaly have
Test 1: LoadGasoline, Assert, InsertKey Assert, StartEngine Assert, Go Assert
Test 2: LoadGasoline, Go, Assert
Test 3: Go, Assert
Test 4: StartEngine, Go, Assert
Depending on the actual object, I would probally not try and do all permutations, but I would have a single test that hits the success track, then I would tests that hit my fringe cases.
Edit:
After some thought I might have tests like:
Start a car key that has no gas
Start a car with gas, and wrong key
Start a car with gas and right key (Test 1 above)
Push Peddle before starting car.
Technically you should use the following tests at least:
testLoadGasoline
testInsertKeyGasolineNotLoaded
testStartEngineKeyNotInserted
testGoEngineNotStarted
testGo
If you can directly view intermediate steps you can add
testInsertKeyGasolineLoaded
testStartEngineKeyInserted
Note that if you can directly set the state (which is language and design dependent), then
testInsertKeyGasolineLoaded might not actually call LoadGasoline.
Related
pytest allows the creation of fixtures that are automatically applied to every test in a test suite (via the autouse keyword argument). This is useful for implementing setup and teardown actions that affect every test case. More details can be found in the pytest documentation.
In theory, the same infrastructure would also be very useful for verifying post-conditions that are expected to exist after each test runs. For example, maybe a log file is created every time a test runs, and I want to make sure it exists when the test ends.
Don't get hung up on the details, but I hope you get the basic idea. The point is that it would be tedious and repetitive to add this code to each test function, especially when autouse fixtures already provide infrastructure for applying this action to every test. Furthermore, fixtures can be packaged into plugins, so my check could be used by other packages.
The problem is that it doesn't seem to be possible to cause a test failure from a fixture. Consider the following example:
#pytest.fixture(autouse=True)
def check_log_file():
# Yielding here runs the test itself
yield
# Now check whether the log file exists (as expected)
if not log_file_exists():
pytest.fail("Log file could not be found")
In the case where the log file does not exist, I don't get a test failure. Instead, I get a pytest error. If there are 10 tests in my test suite, and all of them pass, but 5 of them are missing a log file, I will get 10 passes and 5 errors. My goal is to get 5 passes and 5 failures.
So the first question is: is this possible? Am I just missing something? This answer suggests to me that it is probably not possible. If that's the case, the second question is: is there another way? If the answer to that question is also "no": why not? Is it a fundamental limitation of pytest infrastructure? If not, then are there any plans to support this kind of functionality?
In pytest, a yield-ing fixture has the first half of its definition executed during setup and the latter half executed during teardown. Further, setup and teardown aren't considered part of any individual test and thus don't contribute to its failure. This is why you see your exception reported as an additional error rather than a test failure.
On a philosophical note, as (cleverly) convenient as your attempted approach might be, I would argue that it violates the spirit of test setup and teardown and thus even if you could do it, you shouldn't. The setup and teardown stages exist to support the execution of the test—not to supplement its assertions of system behavior. If the behavior is important enough to assert, the assertions are important enough to reside in the body of one or more dedicated tests.
If you're simply trying to minimize the duplication of code, I'd recommend encapsulating the assertions in a helper method, e.g., assert_log_file_cleaned_up(), which can be called from the body of the appropriate tests. This will allow the test bodies to retain their descriptive power as specifications of system behavior.
AFAIK it isn't possible to tell pytest to treat errors in particular fixture as test failures.
I also have a case where I would like to use fixture to minimize test code duplication but in your case pytest-dependency may be a way to go.
Moreover, test dependencies aren't bad for non-unit tests and be careful with autouse because it makes tests harder to read and debug. Explicit fixtures in test function header give you at least some directions to find executed code.
I prefer using context managers for this purpose:
from contextlib import contextmanager
#contextmanager
def directory_that_must_be_clean_after_use():
directory = set()
yield directory
assert not directory
def test_foo():
with directory_that_must_be_clean_after_use() as directory:
directory.add("file")
If you absoulutely can't afford to add this one line for every test, it's easy enough to write this as a plugin.
Put this in your conftest.py:
import pytest
directory = set()
# register the marker so that pytest doesn't warn you about unknown markers
def pytest_configure(config):
config.addinivalue_line("markers",
"directory_must_be_clean_after_test: the name says it all")
# this is going to be run on every test
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
directory.clear()
yield
if item.get_closest_marker("directory_must_be_clean_after_test"):
assert not directory
And add the according marker to your tests:
# test.py
import pytest
from conftest import directory
def test_foo():
directory.add("foo file")
#pytest.mark.directory_must_be_clean_after_test
def test_bar():
directory.add("bar file")
Running this will give you:
fail.py::test_foo PASSED
fail.py::test_bar FAILED
...
> assert not directory
E AssertionError: assert not {'bar file'}
conftest.py:13: AssertionError
You don't have to use markers, of course, but these allow controlling the scope of the plugin. You can have the markers per-class or per-module as well.
I sometimes see that tests are taged as :pending
ExUnit.start
ExUnit.configure(exclude: :pending)
defmodule SublistTest do
use ExUnit.Case, async: true
test "empty equals empty" do
assert Sublist.compare([], []) == :equal
end
#tag :pending
test "empty is a sublist of anything" do
assert Sublist.compare([], [nil]) == :sublist
end
end
Obviously there are excluded from execution, when you run tests from shell
elixir sublist_test.exs
Is there a way to include :pending test when running tests from command line?
And second question: Why people tag tests as :pending.
You can do this with the mix test task inside a Mix project. Mix projects are super simple to set up:
$ mix new sublist
You can specify your default exclusions in test/test_helper.exs:
ExUnit.start()
ExUnit.configure(exclude: :pending)
then you can write your test in test/sublist_test.exs. To run the tests, do
$ mix test
and to include pending tests as well do
$ mix test --include pending
Now for your second question: people usually mark tests as pending because they are not implemented yet, but they don't want to forget about them. For example you might be on a tight deadline but want to make sure that the tests will eventually be completed. Or maybe the test is not working yet because you need to implement other things first.
If the tests were not excluded by default they would convey the wrong message: that the affected tests are red. But they are rather to do items than actual tests, so they should not fail by default.
I'm applying TDD to my first event centric project (CQRS, Event sourcing etc) and I'm writing my tests according to Greg Young's simple testing framework Given, When, Expect. My test fixture takes a command, commandhandler and aggregate root and then tests the events outputted.
CommandTestFixture<TCommand, TCommandHandler, TAggregateRoot>
For example here is a typical test
[TestFixture]
public class When_moving_a_group :
CommandTestFixture<MoveGroup, MoveGroupHandler, Foo>
I am very happy with these tests on the whole but with the the above test I've hit a problem. The aggregate root contains a collection of groups. The command MoveGroup reorders the collection, taking a from & to index. I setup the test and asserted that the correct GroupMoved event was generated with the correct data.
As an additional test I need to assert that the reordering of the Groups collection actually took place correctly? How do I do this when the aggregate root has no public getters/setters. I could add a method to retrieve the group at a particular index but isn't this breaking encapsulation simply to be testable?
What's the correct way to go about this?
EDIT
The reordering of the groups takes place in the GroupMoved handler on the Aggregate root.
private void Apply(GroupMoved e)
{
var moved = groups[e.From];
groups.RemoveAt(e.From);
groups.Insert(e.To, moved);
}
The friction here comes because you want to assert something about the internal implementation, but what you have at hand is at the top level.
Your tests and assertions need to be at the same logical level. There are two ways to re-arrange this:
What effect does re-ordering groups have on subsequent commands or queries which you do have at the top level?
This should give you an avenue for asserting that the correct outcome occurs without needing to assert anything about the ordering of the groups directly. This keeps the test at the top level and would allow all sorts of internal refactoring (e.g. perhaps lazy sorting of the groups).
Can you test at a lower level?
If you feel that testing as described above is too complicated, you might want to frame your test at a more detailed level. I think of this like focusing in on a section of detail to get it right.
Down at this level (rather than your composite root), the interfaces will know about groups and you'll have the opportunity to assert what you want to assert.
Alternatively, do you need this test at all?
If you can not find a suitable test at either of the above levels, then are you sure you need this test at all? If there is no visible external difference then there is no need to lock the behaviour in place with a test.
I am writing unit tests for a PHP class that maintains users in a database. I now want to test if creating a user works, but also if deleting a user works. I see multiple possibilities to do that:
I only write one method that creates a user and deletes it afterwards
I write two methods. The first one creates the user, saves it's ID. The second one deletes that user with the saved ID.
I write two methods. The first one only creates a user. The second method creates a user so that there is one that can afterwards be deleted.
I have read that every test method should be independent of the others, which means the third possibility is the way to go, but that also means every method has to set up its test data by itself (e.g. if you want to test if it's possible to add a user twice).
How would you do it? What is good unit testing style in this case?
Two different things = Two tests.
Test_DeleteUser() could be in a different test fixture as well because it has a different Setup() code of ensuring that a User already exists.
[SetUp]
public void SetUp()
{
CreateUser("Me");
Assert.IsTrue( User.Exists("Me"), "Setup failed!" );
}
[Test]
public void Test_DeleteUser()
{
DeleteUser("Me");
Assert.IsFalse( User.Exists("Me") );
}
This means that if Test_CreateUser() passes and Test_DeleteUser() doesn't - you know that there is a bug in the section of the code that is responsible for deleting users.
Update: Was just giving some thought to Charlie's comments on the dependency issue - by which i mean if Creation is broken, both tests fail even though Delete. The best I could do was to move a guard check so that Setup shows up in the Errors and Failures tab; to distinguish setup failures (In general cases, setup failures should be easy to spot by an entire test-fixture showing Red.)
How you do this codependent on how you utilize Mocks and stubs. I would go for the more granular approach so having 2 different tests.
Test A
CreateUser("testuser");
assertTrue(CheckUserInDatabase("testuser"))
Test B
LoadUserIntoDB("testuser2")
DeleteUser("testuser2")
assertFalse(CheckUserInDatabase("testuser2"))
TearDown
RemoveFromDB("testuser")
RemoveFromDB("testuser2")
CheckUserInDatabase(string user)
...//Access DAL and check item in DB
If you utilize mocks and stubs you don't need to access the DAL until you do your integration testing so won't need as much work done on the asserting and setting up the data
Usually, you should have two methods but reality still wins over text on paper in the following case:
You need a lot of expensive setup code to create the object to test. This is a code smell and should be fixed but sometimes, you really have no choice (think of some code that aggregates data from several places: You really need all those places). In this case, I write mega tests (where a test case can have thousands of lines of code spread over many methods). It creates the database, all tables, fills them with defined data, runs the code step by step, verifies each step.
This should be a rare case. If you need one, you must actively ignore the rule "Tests should be fast". This scenario is so complex that you want to check as many things as possible. I had a case where I would dump the contents of 7 database tables to files and compare them for each of the 15 SQL updates (which gave me 105 files to compare in a single test) plus about a million asserts that would run.
The goal here is to make the test fail in such a way that you notice the source of the problem right away. It's like pouring all the constraints into code and make them fail early so you know which line of app code to check. The main drawback is that these test cases are hell to maintain. Every change of the app code means that you'll have to update many of the 105 "expected data" files.
I'm somewhat new to writing tests. I've find myself struggling with keeping my setUp's clean and concise, instead trying to accomplish too much with an uber-setUp.
My question is, how do you split up your testing?
Do your tests include one or two lines of independent step code?
def test_public_items():
item1 = PublicItem()
item2 = PublicItem()
assertEqual(public_items, [item1, item2])
or do you factor that into the setUp no matter what?
If that's the case, how do you deal with test class separation? Do you create a new class when one set of tests needs a different setUp then another set of tests?
I believe you've hit a couple of anti-patterns here
Excessive Setup
Inappropriately shared fixture.
The rule of thumb is that all tests in a particular test fixture should need the code in the Setup() method.
If you write a test() that needs more or less setup that what is currently present, it may be a hint that it belongs to a new test fixture. Inertia to create a new test fixture.. is what snowballs the setup code into one big ball of mud - trying to do everything for all tests. Hurts readability quite a bit.. you can't see the test amid the setup code, most of which may not even be relevant for the test you're looking at.
That said it is okay to have a test have some specific setup instructions right in the test over the common setup. (That belongs to the first of the Arrange-Act-Assert triad). However if you have duplication of those instructions in multiple tests - you should probably take all those tests out to a new test fixture, whose
setup_of_new_fixture = old_setup + recurring_arrange_instruction
Yes, a text fixture (embodied in a class) should exactly be a set of tests sharing common needs for set-up and tear-down.
Ideally, a unit test should be named testThisConditionHolds. public_items is not a "condition". Wherever the incredibly-black-magic public_items is supposed to come from I'd be writing tests like:
def testNoPublicItemsRecordedIfNoneDefined(self):
assertEqual([], public_items)
def testOnePublicItemsIsRecordedRight(self):
item = PublicItem()
assertEqual([item], public_items)
def testTwoPublicItemsAreRecordedRight(self):
item1 = PublicItem()
item2 = PublicItem()
assertEqual([item1, item2], public_items)
If public_items is a magical list supposed to be magically populated as a side effect of calling a magic function PublicItem, then calling the latter in setUp would in fact destroy the ability to test these simple cases properly, so of course I wouldn't do it!