I have a large program that I need to write tests for. I'm wondering if it would be wrong to write tests that run in a specific order, as some necessarily have to run in order and that depend upon a previous test.
For example a scenario like the following:
CreateEmployer
CreateEmployee (requires employer)
Add Department
The drawback I see to this approach is that if one test fails, all the of the follow tests will also fail. But, I am going to have to write the code to build the database anyway, so it might be a more effective approach to use the code that builds the mock database as a sort of integration-like unit test.
Should I create the database without using the tests as a seed method, and then run each of the methods again to see the result? The problem I see with this approach is that if the seed method does not work all of the tests will fail and it won't be immediately clear that the error is in the seed method and not the services or the tests themselves.
Yes, this is discouraged. Tests shouldn't be "temporally coupled".
Each test should run in complete isolation of other tests. If you find yourself in a situation where the artifacts created by Test A are needed by Test B then you have two problems to correct:
Test A shouldn't be creating artifacts (or side-effects).
Test B should be using mock data as part of its "arrange" step.
Basically, the unit tests shouldn't be using a real database. They should be testing the logic of the code, not the interaction with the infrastructure dependencies. Mock those dependencies in order to test just the code.
It is a bad idea to have unit tests depend upon each other. In your example you could have a defect in both CreateEmployer and AddDepartment, but when all three fail because of the CreateEmployer test you might mistakenly assume that only the CreateEmployer test is 'really' failing. This means you've lost the potently valuable information that AddDepartment is failing as well.
Another problem is that you might create a separate workflow in the future that calls AddDepartment without calling CreateEmployer. Now your tests assume that CreateEmployer will always be called, but in reality it isn't. All three tests could pass, but the application could still break because you have a dependency you didn't know was there.
The best tests won't rely on a database at all, but will instead allow you to manually specify or "Mock" the data. Then you don't have to worry about an unrelated database problem breaking all of your tests.
If these are truly Unit tests then yes, requiring a specific order is a bad practice for several reasons.
Coupling - As you point out, if one test fails, then subsequent ones will fail as well. This will mask real problems.
TDD - A core principle of TDD is make tests easy to run. If you do so, developers are more likely to run them. If they are hard to run (e.g. I have to run the entire suite), then they are less likely to be run and their value is lost.
Ideally, unit tests should not depend upon the completion of another test in order to run. It is also hard to run them in a given sequence, but that may depend upon your unit testing tool.
In your example, I would create a test that tests the CreateEmployer method and makes sure it returns a new object the way you expect.
The second test I would create would be CreateEmployee, and if that test requires an Employer object, using dependency injection, your CreateEmployee method could receive its Employer object. Here is where you would use a mock object (one that code to get created, returning a fixed/known Employer) as the Employer object the CreateEmployee method would consume. This lets you test the CreateEmployee method and its actions upon that object, with a given/known instance of the Employer object.
Your third test, AddDepartment, I assume also depends upon an Employer object. This unit test can follow the same pattern, and receive a mock Employer object to consumer during its test. (The same object you pass to unit test two above.)
Each test now runs/fails on its own, and can run in any sequence.
Related
Exactly how independent should unit tests be? What should be done in a "before" section of a unit testing suite?
Say, for example, I am testing the functionality of a server - should the server be created, initialised, connected to it's various data sources, &c. inside the body of every test case. Are there situations where it may be appropriate to initialise the server once, and then test more than one case.
The other situation I am considering is mobile app testing - where the phone objects need to be created to perform a unit test. Should this be done every time. Create phone, initialise, run test, destroy phone, repeat?
Unit tests should be completely independent, i.e. each should be able to run in any order so each will need to have its own initialization steps.
Now, if you are talking about server or phone initialization, it sounds more like integration tests rather than unit tests.
Ideally yes. Every test should start from scratch, and put the system into a particular well-defined state before executing the function under test. If you don't, then you make it more difficult to isolate the problem when the test fails. Even worse, you may cause the test not to fail because of some extra state left behind by an earlier test.
If you have a situation where the setup time is too long, you can mock or stub some of the ancillary objects.
If you are worried about having too much setup code, you can refactor the setup code into reusable functions.
I'm currently broadening my Unit Testing by utilising Mock objects (nSubsitute in this particular case). However I'm wondering what the current wisdom when creating a Mock objects. For instance, I'm working with an object that contains various routines to grab and process data - no biggie here but it will be utilised in a fair number of tests.
Should I create a shared function that returns the Mock Object with all the appropriate methods and behaviours mocked for pretty much most of the Testing project and call that object into my Unit Tests? Or shall I Mock the object into every Unit Test, only mocking the behaviour I need for that test (although there will be times I'll be mocking the same behaviour more than one occasion).
Thoughts or advice is gratefully received...
I'm not sure if there is an agreed "current wisdom" on this, but here's my 2 cents.
First, as #codebox pointed out, re-creating your mocks for each unit test is a good idea, as you want your unit tests to run independently of each other. Doing otherwise can result in tests that pass when run together but fail when run in isolation (or vis versa). Creating mocks required for tests is commonly done in test setup ([SetUp] in NUnit, constructor in XUnit), so each test will get a newly created mock.
In terms of configuring these mocks, it depends on the situation and how you test. My preference is to configure them in each test with the minimum amount of configuration necessary. This is a good way of communicating exactly what that test requires of its dependencies. There is nothing wrong with some duplication in these cases.
If a number of tests require the same configuration, I would consider using a scenario-based test fixture (link disclaimer: shameless self-promotion). A scenario could be something like When_the_service_is_unavailable, and the setup for that scenario could configure the mocked service to throw an exception or return an error code. Each test then makes assertions based on that common configuration/scenario (e.g. should display error message, should send email to admin etc).
Another option if you have lots of duplicated bits of configuration is to use a Test Data Builder. This gives you reusable ways of configuring a number of different aspects of your mock or other any other test data.
Finally, if you're finding a large amount of configuration is required it might be worth considering changing the interface of the test dependency to be less "chatty". By looking for a valid abstraction that reduces the number of calls required by the class under test you'll have less to configure in your tests, and have a nice encapsulation of the responsibilities on which that class depends.
It is worth experimenting with a few different approaches and seeing what works for you. Any removal of duplication needs to be balanced with keeping each test case independent, simple, maintainable and reliable. If you find you have a large number of tests fail for small changes, or that you can't figure out the configuration an individual tests needs, or if tests fail depending on the order in which they are run, then you'll want to refine your approach.
I would create new mocks for each test - if you re-use them you may get unexpected behaviour where the state of the mock from earlier tests affects the outcome of later tests.
It's hard to provide a general answer without looking at a specific case.
I'd stick with the same approach as I do everywhere else: first look at the tests as independent beings, then look for similarities and extract the common part out.
Your goal here is to follow DRY, so that your tests are maintainable in case the requirements change.
So...
If it's obvious that every test in a group is going to use the same mock behaviour, provide it in your common set-up
If each of them is significantly different, as in: the content of the mock constitutes a significant part of what you're testing and the test/mock relationship looks like 1:1, then it's reasonable to keep them close to the tests
If the mocks differ between them, but only to some degree, you still want to avoid redundancy. A common SetUp won't help you, but you may want to introduce an utility like PrepareMock(args...) that will cover different cases. This will make your actual test methods free of repetitive set-up, but still let you introduce any degree of difference between them.
The tests look nice when you extract all similarities upwards (to a SetUp or helper methods) so that the only thing that remains in test methods is what's different between them.
This is generic to any language in which unit testing is done.
Most unit test libraries provide a way to control the order in which unit tests are run. Let's say I have a TestClass that defines twelve tests. Is there any good reason to try controlling the order the twelve tests run in? Keep in mind that any startup/shutdown code is already taken care of, because most libraries provide a way to do that to. The advantage I see to having an explicit test order is that you can compose your tests so each one uses only functionality that it tests directly or has already been tested by a prior test. The disadvantage is the maintenance cost of keeping the ordering up to date and ensuring that other developers understand why the order is what it is and work to preserve it.
Is this just not worth the effort?
It's not worth the effort. More importantly than that, it's not a good practice. Each unit test should run independently from the others. If one of your tests depends on another being run first, it's not a good test.
As far as only using functionality that has been tested by another test, you don't need to "order" the tests to achieve this. Let's say you have a piece of basic logic, and there is a test for that logic (Test A). You test a more complex piece of logic in a new Test B, and this new test assumes that the basic logic is working. If something later goes wrong with the basic logic, Test A will fail, and Test B may fail also. That is fine. Test A will pinpoint the problem for you to fix it. It doesn't matter what order the tests run in.
The advantage I see to having an explicit test order is that you can compose your tests so each one uses only functionality that it tests directly or has already been tested by a prior test.
I don't see much advantage here. Also, dependency frequently spreads across classes, so trying to sequence test runs by feature dependency would likely spread across multiple test classes, so ordering the run within a single test class wouldn't cover it.
I am looking for rules like:
A test is not a unit-test if:
it communicates with a database
it cannot run in parallel with other tests
uses the "environment" like registry or file system
What else is there?
See Michael Feathers' definition
A test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing
config files) to run it.
A test is not a unit test if it is not testing a unit.
Seriously, that's all there is to it.
The concept of "unit" in unit testing is not well-defined, in fact, the best definition I have found so far, isn't actually a definition because it is circular: a unit in a unit test is the smallest possible thing that can be tested in isolation.
This gives you two checkpoints: is it tested in isolation? And is it the smallest possible thing?
Please note that both of these are context-dependent. What might be the smallest possible thing in one situation (say, an entire object) might in another situation just one small piece of one single method. And what counts as isolation in one situation might be in another (e.g. in a memory-managed language, you never run in isolation from the garbage collector, and most of the time that is irrelevant, but sometimes it might not be).
Difficult one...
For me a unit test verifies one specific piece of logic in isolation. Meaning, I take some logic, extract it from the rest (if necessary by mocking dependencies) and test just that logic - a unit (of the whole) - by exploring different kind of possible control flows.
But on the other side...can we always 100% say correct or incorrect?? Not to become philosophical, but - as also Michael says in his post:
Tests that do these things aren't bad.
Often they are worth writing, and they
can be written in a unit test harness.
However, it is important to be able to
separate them from true unit tests so
that we can keep a set of tests that
we can run fast whenever we make our
changes.
So why shouldn't I write a unit test that verifies the logic of parsing for instance an xls file by accessing some dummy file from the file system in my test folder (like MS tests allow with the DeploymentItem)?
Of course - as mentioned - we should separate these kind of tests from the others (maybe in a separate test suite in JUnit). But I think one should also write those tests if he feels comfortable in having them there...clearly then always again remembering that a unit test should just test a piece in isolation.
What is most important in my eyes is that these tests run fast and don't take too long s.t. they can be run repeatedly and very often.
It has no asserts, and is not expecting an exception to be thrown.
A test is not an Unit Test when:
it tests more than one thing at once (i.e. it tests how two things work together) - then it is an integration test
Checklist for good unit tests:
they are automated
they are repeatable
they are easy to implement
they remain for future use, once written
they can be run by anyone
they can be run by the push of a button
they run quickly
Some more best practices (in no particular order of importance):
tests should be separated from integration tests (which are slower), so that they can be run fast as frequently as possible
they should not comprise too much logic (preferably, no control structures)
every test should test only one thing (thus, they should contain only one assert)
the expected values used in assertions should be hard-coded and not computed at test run-time
external dependencies (filesystem, time, memory etc.) should be replaced by stubs
test should recreate the initial state at test shutdown
in assertions, it is better to use a "contains..." policy, rather than "is strictly equal..." policy (i.e. we expect certain values in a collection, certain characters in a string etc.)
This is part of the knowledge I have extracted from Roy Osherove's book - The Art of Unit Testing
Implementing a test across multiple possibly failing units would not be a unit test.
Intricate question.
Say I am to program some business logic and all business logic needs to get to the data via some form of DAL.
Say that for the purposes of testing, I mock the DAL units (by creating "mockingbirds").
But those mockingbirds are of course, additional units in their own right. So even when using mocks, it might seem like I'm still bound to violate the idea of "no other units involved" when I want to unit-test my business logic module.
Of course, it is generally known that "creating mockingbirds for the DAL" can invalidate your very test itself on the count that your mockingbird deviates in some particular aspect from the DAL.
Conclusion : it is outright impossible to do "genuine unit-tests" on business modules that depend in any way on any kind of DAL, question mark ?
Corrolary : the only thing that can possible be ("genuinely" !) unit-tested is the DAL itself, question mark ?
Corrolary of the corrolary : given that the "DAL" is usually either an ORM or the very DML of some DBMS, and given that those products are usually bought as being "proven technology", what is the added value of doing any unit tests what so ever, question mark ?
After whether a test is a unit test or not is settled the next question is, is it a good unit test?
I have been creating Unit tests like crazy and find that I'm often having to set up something in one test that I just tore down in a previous test. Is it ever reasonable to create something (e.g. a database record) in one test (e.g. an Insertion test) and then use it for a later test (e.g. a Deletion test)? Or should each and every test always stand completely on its own?
Can you even determine the order of tests in NUnit or are they always done alphabetically?
Note: I am specifically asking about the order of tests within one test file. Not across test files or in any way more globally.
Update: Thanks to everyone that answered - there were a lot of good answers and the sense of the group is pretty unanimous. I've chosen John Nolan's answer as he provided the most complete explanation and lots of links. As you may have guessed, I've been sorely tempted to break this rule despite thinking that it might be a bit "smelly" as John put it. Thanks also to Fortyrunner for adding the unit-testing tag.
Look into test fixture setups that allow you to specify functions that will be executed before any of the tests in the fixture. This allows you to do common setup once and it will always run, whether you run one test, or all tests in the suite.
Relying on the order of your tests indicates that you are persisting state across tests. This is smelly
A cleaner way of testing is where you only depend on the single piece of functionality you want to check the behaviour of. Commonly you mock the other objects you need to get your method under test to function.
A good way to think about approaching unit tests is the Arrange, Act, Assert pattern.
Below is a snippet from Karl Seguin's excellent free eBook. I've annoted Arrange, Act and Assert.
[TestFixture] public class CarTest
{
[Test] public void SaveCarCallsUpdateWhenAlreadyExistingCar()
{
//Arrange
MockRepository mocks = new MockRepository();
IDataAccess dataAccess = mocks.CreateMock<IDataAccess>();
ObjectFactory.InjectStub(typeof(IDataAccess), dataAccess);
//Act
Car car = new Car();
Expect.Call(dataAccess.Save(car)).Return(389);
mocks.ReplayAll();
car.Save();
mocks.VerifyAll();
// Assert
Assert.AreEqual(389, car.Id);
ObjectFactory.ResetDefaults();
}
}
Unit tests are intended to stand alone, not be run as a sequential script. If you actually need them run sequentially, collect them into a single test function.
If your unit tests suffer from expensive set-up, you may be doing integration testing when you think you're doing unit testing. If you're hitting a SQL database inside most of your unit tests, you're actually integration testing with your data access layer.
I would view each test as completely independent from any other test. Even if you could mandate the order of the tests, it would be a maintenance nightmare when the tests must change.
I really wouldn't rely on ordering of tests. Instead, I'd pull the common setup code into a separate method and call that from both the simple test and the more complicated one. Alternatively, just call the insertion test itself at the start of the deletion test.
I would strongly advise to make all your unit tests independent.
Your business logic / database structure etc. may change over time, so that you'll eventually have to replace or rewrite (or even discard) existing unit tests - and if you have several other tests depending on the one that you're replacing, this might cause unnecessary troubles because you'd have to go through all of the other tests as well and check if these are still working as expected.
In addition, one failing unit test should not be able to drag many others (that might perfectly work on their own) down.
Unfortunately run order of the unit test is not predictable or at least could be changed in future. E.g. unit testing framework will be changed so each test will be executed in separate thread.
So from my point of view using test order is not reasonable.
On the other hand you can create a set of small independent tests to test small parts of your code and then create one or several large tests that will run your small tests in specific order.
If you have stateful tests (a common problem with database work - which is what I do when I'm not on SO), then it seems to me that avoiding order within a test file is not absolutely necessary. However, you have to recognize that if you have 2 tests, with test 2 depending on test 1 passing, then you will get a 'catastrophic' double failure if test 1 fails, because test 2 doesn't have the expected setup (and, what's more, you want to worry if test 2 does pass after test 1 failed if you think that test 2 depends on test 1 passing).
That's why you want the tests independent whenever possible - both intra-file and inter-file.
It would be very unwise to depend on the order between (sets of) tests in different files.