This seemed to spark a bit of conversation on another question and I
thought it worthy to spin into its own question.
The DRY principle seems to be our weapon-of-choice for fighting maintenance
problems, but what about the maintenance of test code? Do the same rules of thumb
apply?
A few strong voices in the developer testing community are of the opinion that
setup and teardown are harmful and should be avoided... to name a few:
James Newkirk
Jay Fields, [2]
In fact, xUnit.net has removed them from the framework altogether for this very reason
(though there are ways to get around this self-imposed limitation).
What has been your experience? Do setup/teardown hurt or help test maintainability?
UPDATE: do more fine-grained constructs like those available in JUnit4 or TestNG (#BeforeClass, #BeforeGroups, etc.) make a difference?
The majority (if not all) of valid uses for setup and teardown methods can be written as factory methods which allows for DRY without getting into issues that seem to be plagued with the setup/teardown paradigm.
If you're implementing the teardown, typically this means you're not doing a unit test, but rather an integration test. A lot of people use this as a reason to not have a teardown, but IMO there should be both integration and unit test. I would personally separate them into separate assemblies, but I think a good testing framework should be able to support both types of testing. Not all good testing is going to be unit testing.
However, with the setup there seems to be a number of reasons why you need to do things before a test is actually run. For example, construction of object state to prep for the test (for instance setting up a Dependency Injection framework). This is a valid reason for a setup, but could just as easily be done with a factory.
Also, there is a distinction between class and method level setup/teardown. That needs to be kept in mind when considering what you're trying to do.
My biggest problem that I have had with using the setup/teardown paradigm is that my tests don't always follow the same pattern. This has brought me into using factory patterns instead, which allows me to have DRY while at the same time being readable and not at all confusing to other developers. Going the factory route, I've been able to have my cake and eat it to.
They've really helped with our test maintainability. Our "unit" tests are actually full end-to-end integration tests that write to the DB and check the results. Not my fault, they were like that when I got here, and I'm working to change things.
Anyway, if one test failed, it went on to the next one, trying to enter the same user from the first test in the DB, violating a uniqueness constraint, and the failures just cascaded from there. Moving the user creation/deletion into the [Fixture][SetUp|TearDown] methods allowed us to see the one test that failed without everything going haywire, and made my life a lot easier and less stabby.
I think the DRY principle applies just as much for tests as it does for code, however its application is different. In code you go to much greater lengths to literally not do the same thing in two different parts of the code. In tests the need to do that (do a lot of the same setup) is certainly a smell, but the solution is not necessarily to factor out the duplication into a setup method. It may be make the state easier to set up in the class itself or to isolate the code under test so it is less dependent on this amount of state to be meaningful.
Given the general goal of only testing one thing per test, it really isn't possible to avoid doing a lot of the same thing over and over again in certain cases (such as creating an object of a certain type). If you find you have a lot of that, it may be worth rethinking the test approach, such as introducing parametrized tests and the like.
I think setup and teardown should be primarily for establishing the environment (such as injections to make the environment a test one rather than a production one), and should not contain steps that are part and parcel of the test.
I agree with everything Joseph has to say, especially the part about tearDown being a sign of writing integration tests (and 99% of the time is what I've used it for), but in addition to that I'd say that the use of setup is a good indicator of when tests should be logically grouped together and when they should be split into multiple test classes.
I have no problem with large setup methods when applying tests to legacy code, but the setup should be common to every test in the suite. When you find yourself having the setup method really doing multiple bits of setup, then it's time to split your tests into multiple cases.
Following the examples in "Test Driven", the setup method comes about from removing duplication in the test cases.
I use setup quite frequently in Java and Python, frequently to set up collaborators (either real or test, depending). If the object under test has no constructors or just the collaborators as constructors I will create the object. For a simple value class I usually don't bother with them.
I use teardown very infrequently in Java. In Python it was used more often because I was more likely to change global state (in particular, monkey patching modules to get users of those modules under test). In that case I want a teardown that will guaranteed to be called if a test failed.
Integration tests and functional tests (which often use the xunit framework) are more likely to need setup and teardown.
The point to remember is to think about fixtures, not only DRY.
I don't have an issue with test setup and teardown methods per se.
The issue to me is that if you have a test setup and teardown method, it implies that the same test object is being reused for each test. This is a potential error vector, as if you forget to clean up some element of state between tests, your test results can become order-dependent. What we really want is tests that do not share any state.
xUnit.Net gets rid of setup/teardown, because it creates a new object for each test that is run. In essence, the constructor becomes the setup method, and the finalizer becomes the teardown method. There's no (object-level) state held between tests, eliminating this potential error vector.
Most tests that I write have some amount of setup, even if it's just creating the mocks I need and wiring the object being tested up to the mocks. What they don't do is share any state between tests. Teardown is just making sure that I don't share that state.
I haven't had time to read both of what you posted, but I in particular liked this comment:
each test is forced to do the initialization for what it needs to run.
Setup and tear down are convenience methods - they shouldn't attempt to do much more than initialize a class using its default constructor, etc. Common code that three tests need in a five test class shouldn't appear there - each of the three tests should call this code directly. This also keeps tests from stepping on each other's toes and breaking a bunch of tests just because you changed a common initalization routine. The main problem is that this will be called before all tests - not just specific tests. Most tests should be simple, and the more complex ones will need initialization code, but it is easier to see the simplicity of the simple tests when you don't have to trace through a complex initialization in set up and complex destruction in tear down while thinking about what the test is actually supposed to accomplish.
Personally, I've found setup and teardown aren't always evil, and that this line of reasoning is a bit dogmatic. But I have no problem calling them a
code smell for unit tests. I feel their use should be justified, for a few reasons:
Test code is procedural by its nature. In general, setup/teardown do tend to reduce test readability/focus.
Setup methods tend to initialize more than what is needed for any single test. When abused they can become unwieldy. Object Mothers, Test Data Builders, perhaps frameworks like FactoryGirl seem better at initializing test data.
They encourage "context bloat" - the larger the test context becomes, the less maintainable it will be.
To the extent that my setup/teardown doesn't do this, I think their use is warranted. There will always be some duplication in tests. Neal Ford states this as "Tests can be wet but not soaking..." Also, I think their use is more justified when we're not talking about unit tests specifically, but integration tests more broadly.
Working on my own, this has never really been a problem. But I've found it very difficult to maintain test suites in a team setting, and it tends to be because we don't understand each other's code immediately, or don't want to have to step through it to understand it. From a test perspective, I've found allowing some duplication in tests eases this burden.
I'd love to hear how others feel about this, though.
If you need setup and teardown to make your unit tests work, maybe what you really need is mock objects?
A Unit-Test should
produce deterministic result
be independent
be valid
...
What other characteristics should a test also have?
Ah. My favorite subject :-) Where to start...
According to xUnit test patterns by Gerard Meszaros (THE book to read about unit testing)
Tests should reduce risk, not
introduce it.
Tests should be easy to run.
Tests should be easy to maintain as
the system evolves around them
Some things to make this easier:
Tests should only fail because of one
reason. (Tests should only test one thing, avoid multiple asserts for example.)
There should only be one test that fails for that reason. (this keeps your testbase maintainable)
Minimize test dependencies (no
dependencies on databases, files, ui
etc.)
Other things to look at:
Naming
Have a descriptive name. Tests-names should read like specifications. If your names get too long you're probably testing too much.
Structure
Use AAA structure. This is the new fad for mocking frameworks, But I think it's a good way to structure all your tests like this.
Arrange your context
Act, do the things that need to be tested
Assert, assert what you want to check
I usually divide my tests in three blocks of code. Knowing this pattern makes tests more readable.
Mocks vs. Stubs
When using a mocking framework always try to use stubs and state based testing before resorting to mocking.
Stubs are objects that stand in for dependencies of the object you're trying to test. You can program behaviour into them and they can get called in your tests. Mocks expand on that by letting you assert if they were called and how. Mocking is very powerfull but it lets you test implementation instead of pre and post-conditions of your code. This tends to make tests more brittle.
The Pragmatic Programmers' answer : good tests shall be A-TRIP
Automatic
Thorough
Repeatable
Independent
Professional
not access external resources
be readable
Automatable: no manual intervention should be required to run the tests (CI).
Complete: they must cover as much code they can (Code Coverage).
Reusable: no need to create tests that will only be executed once.
Independent: Independent execution of a test should not affect the performance of another.
Professional: tests should be considered with the same value as the code, the same professionalism, documentation, etc.
One that I haven't seen anyone else mention is small. A unit test should test for one particular thing and that's all. I try to aim to have only one assert and minimize the amount of setup code by refactoring them out into their own methods. I'll also create my own custom asserts. A nice small unit test IMO is about 10 lines or less. When a test is small it is easy to get a quick understanding of what the test is trying to do. Large tests end up being unmaintainable in the long run.
Of course, small isn't the only thing I aim for...its just one of the things I value in a unit test. :-)
An other factors to keep in mind is the running time. If a test runs too long it is likely to be skipped.
Must be fully automatic.
Must not assume any preconditions
(product X be installed, file and Y
location etc).
Must be person independent as far as
running the scripts are concerned. Result can, however, be analysed by
subject experts only.
Must run on every beta build.
Must produce a verifiable report.
A unit test should be fast: hundreds of test should be able to run in a few seconds.
A test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing config files) to run it.
Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.
source: A Set of Unit Testing Rules
I have been creating Unit tests like crazy and find that I'm often having to set up something in one test that I just tore down in a previous test. Is it ever reasonable to create something (e.g. a database record) in one test (e.g. an Insertion test) and then use it for a later test (e.g. a Deletion test)? Or should each and every test always stand completely on its own?
Can you even determine the order of tests in NUnit or are they always done alphabetically?
Note: I am specifically asking about the order of tests within one test file. Not across test files or in any way more globally.
Update: Thanks to everyone that answered - there were a lot of good answers and the sense of the group is pretty unanimous. I've chosen John Nolan's answer as he provided the most complete explanation and lots of links. As you may have guessed, I've been sorely tempted to break this rule despite thinking that it might be a bit "smelly" as John put it. Thanks also to Fortyrunner for adding the unit-testing tag.
Look into test fixture setups that allow you to specify functions that will be executed before any of the tests in the fixture. This allows you to do common setup once and it will always run, whether you run one test, or all tests in the suite.
Relying on the order of your tests indicates that you are persisting state across tests. This is smelly
A cleaner way of testing is where you only depend on the single piece of functionality you want to check the behaviour of. Commonly you mock the other objects you need to get your method under test to function.
A good way to think about approaching unit tests is the Arrange, Act, Assert pattern.
Below is a snippet from Karl Seguin's excellent free eBook. I've annoted Arrange, Act and Assert.
[TestFixture] public class CarTest
{
[Test] public void SaveCarCallsUpdateWhenAlreadyExistingCar()
{
//Arrange
MockRepository mocks = new MockRepository();
IDataAccess dataAccess = mocks.CreateMock<IDataAccess>();
ObjectFactory.InjectStub(typeof(IDataAccess), dataAccess);
//Act
Car car = new Car();
Expect.Call(dataAccess.Save(car)).Return(389);
mocks.ReplayAll();
car.Save();
mocks.VerifyAll();
// Assert
Assert.AreEqual(389, car.Id);
ObjectFactory.ResetDefaults();
}
}
Unit tests are intended to stand alone, not be run as a sequential script. If you actually need them run sequentially, collect them into a single test function.
If your unit tests suffer from expensive set-up, you may be doing integration testing when you think you're doing unit testing. If you're hitting a SQL database inside most of your unit tests, you're actually integration testing with your data access layer.
I would view each test as completely independent from any other test. Even if you could mandate the order of the tests, it would be a maintenance nightmare when the tests must change.
I really wouldn't rely on ordering of tests. Instead, I'd pull the common setup code into a separate method and call that from both the simple test and the more complicated one. Alternatively, just call the insertion test itself at the start of the deletion test.
I would strongly advise to make all your unit tests independent.
Your business logic / database structure etc. may change over time, so that you'll eventually have to replace or rewrite (or even discard) existing unit tests - and if you have several other tests depending on the one that you're replacing, this might cause unnecessary troubles because you'd have to go through all of the other tests as well and check if these are still working as expected.
In addition, one failing unit test should not be able to drag many others (that might perfectly work on their own) down.
Unfortunately run order of the unit test is not predictable or at least could be changed in future. E.g. unit testing framework will be changed so each test will be executed in separate thread.
So from my point of view using test order is not reasonable.
On the other hand you can create a set of small independent tests to test small parts of your code and then create one or several large tests that will run your small tests in specific order.
If you have stateful tests (a common problem with database work - which is what I do when I'm not on SO), then it seems to me that avoiding order within a test file is not absolutely necessary. However, you have to recognize that if you have 2 tests, with test 2 depending on test 1 passing, then you will get a 'catastrophic' double failure if test 1 fails, because test 2 doesn't have the expected setup (and, what's more, you want to worry if test 2 does pass after test 1 failed if you think that test 2 depends on test 1 passing).
That's why you want the tests independent whenever possible - both intra-file and inter-file.
It would be very unwise to depend on the order between (sets of) tests in different files.
I do TDD, and I've been fairly loose in organizing my unit tests. I tend to start with a file representing the next story or chunk of functionality and write all the unit-tests to make that work.
Of course, if I'm introducing a new class, I usually make a separate unit-test module or file for that class, but I don't organize the tests themselves into any higher level structure. The result is I write code fast and I believe my actual program is reasonably well structured, but the unit tests themselves are "messy". Especially, their structure tends to recapitulate the phylogeny of the development process. Sometimes I see myself as trading laziness in the code for laziness in the tests.
How big a problem is this? Who here continually refactors and reorganizes their unit tests to try to improve their overall structure? Any tips for this? What should the overall structure of tests look like.
(Note, that I'm not so much asking the "how many assertions per function" question asked here : How many unit tests should I write per function/method? I'm talking about the bigger picture.)
Divide your tests in 2 sets:
functional tests
units tests
Functional tests are per-user story. Unit tests are per-class. The former check that you actually support the story, the latter exercise and document your functionality.
There is one directory (package) for functional tests. Unit tests should be closely bound with functionality they exercise (so they're scattered). You move them around and refactor them as you move & refactor your code around.
The less important part is organizing the tests.
I start by putting the tests into a class that relates to the class under test, so com.jeffreyfredrick.Foo has a test com.jeffreyfredrick.FooTest. But if some subset of those classes need a different setup then I'll move them into their own test class. I put my tests into a separate source directory but keep them in the same project.
The more important part is refactoring the tests.
Yes I try and refactor my tests as I go. The goal is to remove duplication while still remaining declarative and easy to read. This is true both within test classes and across test classes. Within a test class I might have a parametrized method for creating a test fake (mock or stub). My test fakes are usually inner classes within a test class but if I find there's need I'll pull them out for reuse across tests. I'll also create a TestUtil class with common methods when it seems appropriate.
I think refactoring yours tests is important to long term success of unit testing on large projects. Have you ever heard people complaining about how their tests are too brittle or preventing them from change? You don't want to be in a position where changing the behavior of a class means making dozens or even hundreds of changes to your tests. And just like with code, you achieve this through refactoring and keeping the tests clean.
Tests are code.
I write a unit test class for each class in the application, and keep the test classes organized in the same package structure as the classes under test.
Inside each test class I don't really have much organizational structure. Each one only has a handful of methods for each public method in the class under test, so I've never had any problem finding what I'm looking for.
For every class in the software, I maintain a unit test class. The unit test classes follow the same package hierarchy as the classes which are tested.
I keep my unit test code in a separate project. Some people also prefer to keep their test code in the same project under a separate source directory called 'test'. You could follow whatever feels comfortable to you.
I try to look at the unit tests as a project on their own. As with any project the organisation should follow some internal logic. It does not however have to be specific or formally defined - anything you're comfortable with is OK as long as it keeps your project well-organised and clean.
So for the unit tests I usually either follow the main project code structure or (sometimes when the situation calls of it) focus on the functional areas instead.
Leaving them in one heap is as you might imagine messy and difficult to maintain
I know the so-called textbook definition of unit tests and integration tests. What I am curious about is when it is time to write unit tests... I will write them to cover as many sets of classes as possible.
For example, if I have a Word class, I will write some unit tests for the Word class. Then, I begin writing my Sentence class, and when it needs to interact with the Word class, I will often write my unit tests such that they test both Sentence and Word... at least in the places where they interact.
Have these tests essentially become integration tests because they now test the integration of these 2 classes, or is it just a unit test that spans 2 classes?
In general, because of this uncertain line, I will rarely actually write integration tests... or is my using the finished product to see if all the pieces work properly the actual integration tests, even though they are manual and rarely repeated beyond the scope of each individual feature?
Am I misunderstanding integration tests, or is there really just very little difference between integration and unit tests?
The key difference, to me, is that integration tests reveal if a feature is working or is broken, since they stress the code in a scenario close to reality. They invoke one or more software methods or features and test if they act as expected.
On the opposite, a Unit test testing a single method relies on the (often wrong) assumption that the rest of the software is correctly working, because it explicitly mocks every dependency.
Hence, when a unit test for a method implementing some feature is green, it does not mean the feature is working.
Say you have a method somewhere like this:
public SomeResults DoSomething(someInput) {
var someResult = [Do your job with someInput];
Log.TrackTheFactYouDidYourJob();
return someResults;
}
DoSomething is very important to your customer: it's a feature, the only thing that matters. That's why you usually write a Cucumber specification asserting it: you wish to verify and communicate the feature is working or not.
Feature: To be able to do something
In order to do something
As someone
I want the system to do this thing
Scenario: A sample one
Given this situation
When I do something
Then what I get is what I was expecting for
No doubt: if the test passes, you can assert you are delivering a working feature. This is what you can call Business Value.
If you want to write a unit test for DoSomething you should pretend (using some mocks) that the rest of the classes and methods are working (that is: that, all dependencies the method is using are correctly working) and assert your method is working.
In practice, you do something like:
public SomeResults DoSomething(someInput) {
var someResult = [Do your job with someInput];
FakeAlwaysWorkingLog.TrackTheFactYouDidYourJob(); // Using a mock Log
return someResults;
}
You can do this with Dependency Injection, or some Factory Method or any Mock Framework or just extending the class under test.
Suppose there's a bug in Log.DoSomething().
Fortunately, the Gherkin spec will find it and your end-to-end tests will fail.
The feature won't work, because Log is broken, not because [Do your job with someInput] is not doing its job. And, by the way, [Do your job with someInput] is the sole responsibility for that method.
Also, suppose Log is used in 100 other features, in 100 other methods of 100 other classes.
Yep, 100 features will fail. But, fortunately, 100 end-to-end tests are failing as well and revealing the problem. And, yes: they are telling the truth.
It's very useful information: I know I have a broken product. It's also very confusing information: it tells me nothing about where the problem is. It communicates me the symptom, not the root cause.
Yet, DoSomething's unit test is green, because it's using a fake Log, built to never break. And, yes: it's clearly lying. It's communicating a broken feature is working. How can it be useful?
(If DoSomething()'s unit test fails, be sure: [Do your job with someInput] has some bugs.)
Suppose this is a system with a broken class:
A single bug will break several features, and several integration tests will fail.
On the other hand, the same bug will break just one unit test.
Now, compare the two scenarios.
The same bug will break just one unit test.
All your features using the broken Log are red
All your unit tests are green, only the unit test for Log is red
Actually, unit tests for all modules using a broken feature are green because, by using mocks, they removed dependencies. In other words, they run in an ideal, completely fictional world. And this is the only way to isolate bugs and seek them. Unit testing means mocking. If you aren't mocking, you aren't unit testing.
The difference
Integration tests tell what's not working. But they are of no use in guessing where the problem could be.
Unit tests are the sole tests that tell you where exactly the bug is. To draw this information, they must run the method in a mocked environment, where all other dependencies are supposed to correctly work.
That's why I think that your sentence "Or is it just a unit test that spans 2 classes" is somehow displaced. A unit test should never span 2 classes.
This reply is basically a summary of what I wrote here: Unit tests lie, that's why I love them.
When I write unit tests I limit the scope of the code being tested to the class I am currently writing by mocking dependencies. If I am writing a Sentence class, and Sentence has a dependency on Word, I will use a mock Word. By mocking Word I can focus only on its interface and test the various behaviors of my Sentence class as it interacts with Word's interface. This way I am only testing the behavior and implementation of Sentence and not at the same time testing the implementation of Word.
Once I've written the unit tests to ensure Sentence behaves correctly when it interacts with Word based on Word's interface, then I write the integration test to make sure that my assumptions about the interactions were correct. For this I supply the actual objects and write a test that exercises a feature that will end up using both Sentence and Word.
My 10 bits :D
I was always told that Unit Tests is the testing of an individual component - which should be exercised to its fullest. Now, this tends to have many levels, since most components are made of smaller parts. For me, a unit is a functional part of the system. So it has to provide something of value (i.e. not a method for string parsing, but a HtmlSanitizer perhaps).
Integration Tests is the next step up, its taking one or more components and making sure they work together as they should.. You are then above the intricacies of worry about how the components work individually, but when you enter html into your HtmlEditControl , it somehow magically knows wether its valid or not..
Its a real movable line though.. I'd rather focus more on getting the damn code to work full stop ^_^
In unit test you test every part isolated:
in integration test you test many modules of your system:
and this what happens when you only use unit tests (generally both windows are working, unfortunately not together):
Sources:
source1
source2
Unit tests use mocks
The thing you're talking about are integration tests that actually test the whole integration of your system. But when you do unit testing you should actually test each unit separately. Everything else should be mocked. So in your case of Sentence class, if it uses Word class, then your Word class should be mocked. This way, you'll only test your Sentence class functionality.
I think when you start thinking about integration tests, you are speaking more of a cross between physical layers rather than logical layers.
For example, if your tests concern itself with generating content, it's a unit test: if your test concerns itself with just writing to disk, it's still a unit test, but once you test for both I/O AND the content of the file, then you have yourself an integration test. When you test the output of a function within a service, it's a unit-test, but once you make a service call and see if the function result is the same, then that's an integration test.
Technically you cannot unit test just-one-class anyway. What if your class is composed with several other classes? Does that automatically make it an integration test? I don't think so.
using Single responsibility design, its black and white. More than 1 responsibility, its an integration test.
By the duck test (looks, quacks, waddles, its a duck), its just a unit test with more than 1 newed object in it.
When you get into mvc and testing it, controller tests are always integration, because the controller contains both a model unit and a view unit. Testing logic in that model, I would call a unit test.
In my opinion the answer is "Why does it matter?"
Is it because unit tests are something you do and integration tests are something you don't? Or vice versa? Of course not, you should try to do both.
Is it because unit tests need to be Fast, Isolated, Repeatable, Self-Validating and Timely and integration tests should not? Of course not, all tests should be these.
It is because you use mocks in unit tests but you don't use them in integration tests? Of course not. This would imply that if I have a useful integration test I am not allowed to add a mock for some part, fear I would have to rename my test to "unit test" or hand it over to another programmer to work on.
Is it because unit tests test one unit and integration tests test a number of units? Of course not. Of what practical importance is that? The theoretical discussion on the scope of tests breaks down in practice anyway because the term "unit" is entirely context dependent. At the class level, a unit might be a method. At an assembly level, a unit might be a class, and at the service level, a unit might be a component.
And even classes use other classes, so which is the unit?
It is of no importance.
Testing is important, F.I.R.S.T is important, splitting hairs about definitions is a waste of time which only confuses newcomers to testing.
The nature of your tests
A unit test of module X is a test that expects (and checks for) problems only in module X.
An integration test of many modules is a test that expects problems that arise from the cooperation between the modules so that these problems would be difficult to find using unit tests alone.
Think of the nature of your tests in the following terms:
Risk reduction: That's what tests are for. Only a combination of unit tests and integration tests can give you full risk reduction, because on the one hand unit tests can inherently not test the proper interaction between modules and on the other hand integration tests can exercise the functionality of a non-trivial module only to a small degree.
Test writing effort: Integration tests can save effort because you may then not need to write stubs/fakes/mocks. But unit tests can save effort, too, when implementing (and maintaining!) those stubs/fakes/mocks happens to be easier than configuring the test setup without them.
Test execution delay: Integration tests involving heavyweight operations (such as access to external systems like DBs or remote servers) tend to be slow(er). This means unit tests can be executed far more frequently, which reduces debugging effort if anything fails, because you have a better idea what you have changed in the meantime. This becomes particularly important if you use test-driven development (TDD).
Debugging effort: If an integration test fails, but none of the unit tests does, this can be very inconvenient, because there is so much code involved that may contain the problem. This is not a big problem if you have previously changed only a few lines -- but as integration tests run slowly, you perhaps did not run them in such short intervals...
Remember that an integration test may still stub/fake/mock away some of its dependencies.
This provides plenty of middle ground between unit tests and system tests (the most comprehensive integration tests, testing all of the system).
Pragmatic approach to using both
So a pragmatic approach would be: Flexibly rely on integration tests as much as you sensibly can and use unit tests where this would be too risky or inconvenient.
This manner of thinking may be more useful than some dogmatic discrimination of unit tests and integration tests.
Unit Testing is a method of testing that verifies the individual units of source code are working properly.
Integration Testing is the phase of software testing in which individual software modules are combined and tested as a group.
Wikipedia defines a unit as the smallest testable part of an application, which in Java/C# is a method. But in your example of Word and Sentence class I would probably just write the tests for sentence since I would likely find it overkill to use a mock word class in order to test the sentence class. So sentence would be my unit and word is an implementation detail of that unit.
I think I would still call a couple of interacting classes a unit test provided that the unit tests for class1 are testing class1's features, and the unit tests for class2 are testing its features, and also that they are not hitting the database.
I call a test an integration test when it runs through most of my stack and even hits the database.
I really like this question, because TDD discussion sometimes feels a bit too purist to me, and it's good for me to see some concrete examples.
I do the same - I call them all unit tests, but at some point I have a "unit test" that covers so much I often rename it to "..IntegrationTest" - just a name change only, nothing else changes.
I think there is a continuation from "atomic tests" (testing one tiny class, or a method) to unit tests (class level) and integration tests - and then functional test (which are normally covering a lot more stuff from the top down) - there doesn't seem to be a clean cut off.
If your test sets up data, and perhaps loads a database/file etc, then perhaps its more of an integration test (integration tests I find use less mocks and more real classes, but that doesn't mean you can't mock out some of the system).
Integration tests: Database persistence is tested.
Unit tests: Database access is mocked. Code methods are tested.
Unit testing is testing against a unit of work or a block of code if you like. Usually performed by a single developer.
Integration testing refers to the test that is performed, preferably on an integration server, when a developer commits their code to a source control repository. Integration testing might be performed by utilities such as Cruise Control.
So you do your unit testing to validate that the unit of work you have built is working and then the integration test validates that whatever you have added to the repository didn't break something else.
Simple Explanation with Analogies
This answer will focus purely on examples.
Integration Tests
Integration tests check if everything is working together.
Unit Tests
They tell you whether one specific thing is working.
Examples
Consider a car:
Integration test for a car: e.g. does the car drive to Pondicherry and back? If so, the car as whole is working. If it fails, you won't really where where: radiator, transmission, engine, or carburettor?
Unit test for a car: Is the engine is working? This tests just the engine; nothing else. If this test fails, then you can be confident that there is a bug in the engine....This ties in closely with the concept of "fakes". You might need some keys in order to start the engine - except, you don't want to go to the hassle of actually an ignition (with a lock)...instead, you would hotwire the car to start it....in other words you would use a "fake" key.
Similarly, in unit testing, you would use "fakes" in order to make the engine work a particular way. And then you could simply test: "is it running".
I call unit tests those tests that white box test a class. Any dependencies that class requires is replaced with fake ones (mocks).
Integration tests are those tests where multiple classes and their interactions are tested at the same time. Only some dependencies in these cases are faked/mocked.
I wouldn't call Controller's integration tests unless one of their dependencies is a real one (i.e. not faked) (e.g. IFormsAuthentication).
Separating the two types of tests is useful for testing the system at different levels. Also, integration tests tend to be long lived, and unit tests are supposed to be quick. The execution speed distinction means they're executed differently. In our dev processes, unit tests are run at check-in (which is fine cos they're super quick), and integration tests are run once/twice per day. I try and run integration tests as often as possible, but usually hitting the database/writing to files/making rpc's/etc slows.
That raises another important point, unit tests should avoid hitting IO (e.g. disk, network, db). Otherwise they slow down alot. It takes a bit of effort to design these IO dependencies out - i can't admit I've been faithful to the "unit tests must be fast" rule, but if you are, the benefits on a much larger system become apparent very quickly.
A little bit academic this question, isn't it? ;-)
My point of view:
For me an integration test is the test of the whole part, not if two parts out of ten are going together.
Our integration test shows, if the master build (containing 40 projects) will succeed.
For the projects we have tons of unit tests.
The most important thing concerning unit tests for me is, that one unit test must not be dependent on another unit test. So for me both test you describe above are unit tests, if they are independent. For integration tests this need not to be important.
Have these tests essentially become integration tests because they now test the integration of these 2 classes? Or is it just a unit test that spans 2 classes?
I think Yes and Yes. Your unit test that spans 2 classes became an integration test.
You could avoid it by testing Sentence class with mock implementation - MockWord class, which is important when those parts of system are large enough to be implemented by different developers. In that case Word is unit tested alone, Sentence is unit tested with help of MockWord, and then Sentence is integration-tested with Word.
Exaple of real difference can be following
1) Array of 1,000,000 elements is easily unit tested and works fine.
2) BubbleSort is easily unit tested on mock array of 10 elements and also works fine
3) Integration testing shows that something is not so fine.
If these parts are developed by single person, most likely problem will be found while unit testing BubbleSoft just because developer already has real array and he does not need mock implementation.
In addition, it's important to remember that both unit tests and integration tests can be automated and written using, for example, JUnit.
In JUnit integration tests, one can use the org.junit.Assume class to test the availability of environment elements (e.g., database connection) or other conditions.
I get asked this a lot in interviews. Until now I'd ramble on pretentiously about my expertise and pontificate about component and acceptance testing.
For years I'd understood only integration and unit tests. I could, but didn't always bother to, write unit tests as a solo developer honing my skills.
Unit tests
That is a crucial difference. Unit tests are easy to implement and execute, requiring, ideally, no dependencies. That is what mocks are for. It is often easier to not mock everything, particularly where you gain coverage of other functions you wrote. Easier, maybe, but that isn't the idea of unit testing.
I'll reiterate, unit tests are meant to be easy to run and small. Their failure provides immediate insight into where a bug has been introduced.
Here is the hierarchy of tests, from cheap and plentiful at the bottom to slow, expensive, and few, at the top:
Several more layers can be conceptualised, but were omitted for clarity.
Integration tests
With integration tests you would consider bringing in serious external dependencies, such as VMs, virtual networks and appliances. Possibly you could use actual modems, routers, and firewalls where the expense was justified.
These wouldn't be run locally but on a build server. A mixture of local Jenkins and cloud based CI providers fulfil this need.
Other test terminology
That is my understanding that has served me for several years in industry. We could talk about component tests, and get a definition, but if the definition isn't in common circulation then it loses value.
Acceptance tests were what we would call business unit or customer requirements. These would lead the direction of everything and sit at the top of the pyramid (picture a dollar sign).
E2E, or end to end testing was used synonymously with integration tests, but I noticed online it is placed above. I guess it could have more relevance to acceptance tests the integration tests, which would tend to be more detailed with less interest from stakeholders (though immense interest internally in the department).
If you're a TDD purist, you write the tests before you write production code. Of course, the tests won't compile, so you first make the tests compile, then make the tests pass.
You can do this with unit tests, but you can't with integration or acceptance tests. If you tried with an integration test, nothing would ever compile until you've finished!