My business web application (PHP with HTML/Javascript) has lots of very different options (about 1000) which are stored in the database, so the users can change them theirselves. These options for example define if a button, tab or inputfield is visible, the validation of inputs and the workflow, like when e-mails should be sent. Each user has a user-role, which also defines what they're able to see and do.
My users can use any combination of these options, so I find it very difficult to write tests for all these situations. I have 100+ customers so writing tests for each customer is definetely not an option.
The problem is that some options work together. So while testing one option it's necessary to know the value of some other options. Ideally the tests should also be able to read the options-profiles for each customer. But that would almost be like rewriting the whole application, just for testing, which seems error-prone by itself.
Is it common in unit testing to read the database to get the test-data and options, or is that not a good idea?
How would you handle the situation I described ?
First of all yes, that's perfectly possible. Although it's not recommended to write unit tests after the application is already written and its extremely difficult.
Here are a few advises for your case:
Data Providers
Data Providers are making it possible to call the same test with different parameters, which prevents code duplication in your tests. Perfect if you want to test the same method with different configurations.
https://phpunit.de/manual/3.7/en/writing-tests-for-phpunit.html#writing-tests-for-phpunit.data-providers
Mock objects
If objects depend on other objects you use mock objects. Mocking an object is basically nothing more than creating a dummy object which has a defined behavior and won't do anything else than the things you told it to do.
Note that you can also mock the tested class itself! A mock will keep the methods of the mocked class by default, so you can mock the class you want to test and define a specific behavior for some methods while testing another.
If this is still not enough, you might want to think about splitting up you methods in smaller, more specific methods to get smaller units.
https://phpunit.de/manual/3.7/en/test-doubles.html
Keep it small
Unit tests are called unit tests because they test the smallest possible unit of your code without executing anything else. So instead of testing if a button behaves the way it should, just test if its visible if it should be. Just test one kind of behavior and nothing more.
Don't read the database
It is highly unusual to read the database when writing unit tests and its even more unusual to use actual user data. Instead you define test data. Instead of testing your users configuration, you should test every possible configuration
Code Coverage
A decent way to check if your code is covered by the tests is code coverage. It will show you how much and which code was executed by the tests. Although 100% coverage does not mean full coverage in reality, specially in your case. Just because all lines of code were executed does not mean every option was considered. But its a handy tool anyways and you can see which code you are done with and which you forgot about.
https://phpunit.de/manual/current/en/code-coverage-analysis.html
Conclusion:
What you are trying to do is error-prone itself, yes. Because usually you'd write all your tests before writing the actual method. And you will probably write more test code than the application has itself, but that's not uncommon.
I've been trying to learn how to properly unittest and set up unit tests for all of my code on a new project. The project I'm currently doing this for requires me running a lot of actions against Google BigQuery (i.e. create tables, insert, query, delete). I'm feeling like I can't truly test all of this functionality by mocking BigQuery because the actions I do against it are complicated and interdependent, and if there's a break in the middle somewhere, I want to catch it. Is it generally frowned upon to have something like an environment variable that specifies a test account built into my unit tests so they actually run against the remote service? This feels like the best way to truly test everything and hit tests that I couldn't hit with a mock. So, is this something people do? Are there some major downsides to doing things this way?
I tend to have a mix of unit and integration tests in my project. I believe both are equally valuable, but one thing to keep in mind when doing integration testing is to ensure that the tests are stable and repeatable.
There are several approaches, but I favor the approach of making the tests self sufficient by ensuring that all data dependencies are built in the test itself. This is important since you avoid failing tests due to failed assumptions about existing data in your data source.
A variation on this is to have a scaffolding script populate your data source with fixed test data. I find this to be less manageable since it can introduce dependencies between tests and changing the test data for one test may cause failure in another.
What you're looking to do is technically called integration tests but I do see your point. I myself am doing both as well currently. My interaction in my integration tests is with a database. I find that these integration tests often catch way more errors than true unit tests and are generally more beneficial. I will say however that unit tests are important as well.
I have found that integration tests can tend to take a much longer time since it's doing all this interaction and if this is a part of your nightly build process for example this can greatly increase the amount of time it takes for a build to complete. Some of our builds take close to an hour at this point to complete which is sometimes a problem for us.
I will say when you introduce things like environment variables into the mix you have to start making sure that every developer on the team has this environment variable if they want to run the tests. As a general rule of thumb I try to make it as simple as possible for everyone to build and run tests directly out of source control. There is nothing more frustrating than not being able to build source code or execute unit tests directly out of source control.
It's helpful to think of things like BigQuery as just implementation details; means to an end.
Something in your application currently says "I need x - I'll use BigQuery to get it." Instead of having explicit knowledge of BigQuery, this thing could instead have knowledge of "some entity capable of getting x". This is the location of a seam, and is where mocking would take place.
You mentioned that you don't want to mock all of the objects involved in creating a BigQuery request. You are absolutely right in avoiding this. That doesn't mean that you can't mock out BigQuery, though; you just need to move up a rung.
When writing unit tests that deal with XML (e.g. test a class that reads/generates XML) I used to write my asserted outcome XML-String / my input XML String in separate files right next to my unit test. Let's say I have a class "MyTransformer" that transformes one XML format into another. Then I would create three files all in the same package:
MyTransformerTest.java
MyTransformerTestSampleInput.xml
MyTransformerTestExpectedOutput.xml
Then my assertion might look like this (simplified pseudo code for reasons of simplicity):
Reader transformed = MyTransformer.transform(getResourceAsStream("MyTransformerTestSampleInput.xml")));
Reader expected = getResourceAsStream("MyTransformerTestExpectedOutput.xml");
assertXMLEqual(expected, transformed);
However a colleague told me that the file access that I have in this unit test is unacceptable. He proposed creating a literal string constant (private static final String) containing my XML file contents, possibly in a separate groovy class because of the benefit of multi line strings rather than writing the XML file into files.
I dislike the idea of the literal string constants, because even if I have multi line strings in groovy, I still loose syntax highlighting and all the other helpful features of my XML editor that tell me right away if my XML has syntax errors etc.
What do you think? Is the file access really bad? If so: Why? If not why is it ok?
Two problems with files in unit tests:
they slow down the testing cycle. You may have thousands of unit tests which, preferably, get run on every build - so they should be as fast as possible. If you can speed them up (eg, by getting rid of I/O operations) you'd want to do that. Surely it's not always feasible, so you normally separate out the "slow" tests via NUnit [Category] or something similar - and then run those special tests less frequently - say, only on Nightly builds.
they introduce additional dependencies. If a test requires a file, it will fail not only when the logic behind the test is wrong, but also when the file is missing, or test runner doesn't have read permissions etc. Which makes debugging and fixing not so pleasing!
That said, I won't be too strict about not using files in the tests. If possible, try to avoid them but don't get mad. Make sure you consider maintainability vs speed - the cleaner the test, the easier it will be to fix and understand it later.
If your Unit Tests access the files to feed the fake test data into the System Under Test, so you can run tests, that's not a problem. That actually helps your to have wider variety of test data to exercise within the system under test.
However if your System Under Test access the file system, when executing from the a test, that's not a Unit Test. That's an Integration Test. This is because you are accessing cross cutting concerns such as file system, and they cannot categorised as Unit Tests.
You would really isolate/fake out the file access and test the behaviour of your code (if any), using Unit Tests. They are faster and easier to run. It gives your a pin point feedback if written correctly.
In these cases, I have a unit test that uses an internal representation of the file, which is a string literal in this case.
I also will have an Integration Test, to test the code works correctly when writing to the file.
So it is all down to the Unit / Integration test definitions. Both are valid tests, just depends which test you are writing at the time.
If the xml is more readable, or easier to work with in a file, and you have a lot of these tests, I would leave them.
Strictly speaking, unit tests should not use the file system because it is slow. However, readability is more important. XML in a file is easier to read, and can be loaded in an XML friendly editor.
If the tests take to long to run (cause you have a lot of them), or your colleagues complain, move them to integration tests.
If you work in Windows and LINUX, you have to be careful that the files are picked up by your build server.
There are no perfect answers.
Recently we have been adding automated tests to our existing java applications.
What we have
The majority of these tests are integration tests, which may cover a stack of calls like:-
HTTP post into a servlet
The servlet validates the request and calls the business layer
The business layer does a bunch of stuff via hibernate etc and updates some database tables
The servlet generates some XML, runs this through XSLT to produce response HTML.
We then verify that the servlet responded with the correct XML and that the correct rows exist in the database (our development Oracle instance). These rows are then deleted.
We also have a few smaller unit tests which check single method calls.
These tests are all run as part of our nightly (or adhoc) builds.
The Question
This seems good because we are checking the boundaries of our system: servlet request/response on one end and database on the other. If these work, then we are free to refactor or mess with anything inbetween and have some confidence that the servlet under test continues to work.
What problems are we likely to run into with this approach?
I can't see how adding a bunch more unit tests on individual classes would help. Wouldn't that make it harder to refactor as it's much more likely we will need to throw away and re-write tests?
Unit tests localize failures more tightly. Integration-level tests more closely correspond to user requirements and so are better predictor of delivery success. Neither of them is much good unless built and maintained, but both of them are very valuable if properly used.
(more...)
The thing with units tests is that no integration level test can exercise all the code as much as a good set of unit tests can. Yes, that can mean that you have to refactor the tests somewhat, but in general your tests shouldn't depend on the internals so much. So, lets say for example that you have a single function to get a power of two. You describe it (as a formal methods guy, I'd claim you specify it)
long pow2(int p); // returns 2^p for 0 <= p <= 30
Your test and your spec look essentially the same (this is sort of pseudo-xUnit for illustration):
assertEqual(1073741824,pow2(30);
assertEqual(1, pow2(0));
assertException(domainError, pow2(-1));
assertException(domainError, pow2(31));
Now your implementation can be a for loop with a multiple, and you can come along later and change that to a shift.
If you change the implementation so that, say, it's returning 16 bits (remember that sizeof(long) is only guaranteed to be no less than sizeof(short)) then this tests will fail quickly. An integration-level test should probably fail, but not certainly, and it's just as likely as not to fail somewhere far downstream of the computation of pow2(28).
The point is that they really test for diferent situations. If you could build sufficiently details and extensive integration tests, you might be able to get the same level of coverage and degree of fine-grained testing, but it's probably hard to do at best, and the exponential state-space explosion will defeat you. By partitioning the state space using unit tests, the number of tests you need grows much less than exponentially.
You are asking pros and cons of two different things (what are the pros and cons of riding a horse vs riding a motorcycle?)
Of course both are "automated tests" (~riding) but that doesn't mean that they are alternative (you don't ride a horse for hundreds of miles, and you don't ride a motorcycle in closed-to-vehicle muddy places)
Unit Tests test the smallest unit of the code, usually a method. Each unit test is closely tied to the method it is testing, and if it's well written it's tied (almost) only with that.
They are great to guide the design of new code and the refactoring of existing code. They are great to spot problems long before the system is ready for integration tests. Note that I wrote guide and all the Test Driven Development is about this word.
It does not make any sense to have manual Unit Tests.
What about refactoring, which seems to be your main concern? If you are refactoring just the implementation (content) of a method, but not its existence or "external behavior", the Unit Test is still valid and incredibly useful (you cannot imagine how much useful until you try).
If you are refactoring more aggressively, changing methods existence or behavior, then yes, you need to write a new Unit Test for each new method, and possibly throw away the old one. But writing the Unit Test, especially if you write it before the code itself, will help to clarify the design (i.e. what the method should do, and what it shouldn't) without being confused by the implementation details (i.e. how the method should do the thing that it needs to do).
Automated Integration Tests test the biggest unit of the code, usually the entire application.
They are great to test use cases which you don't want to test by hand. But you can also have manual Integration Tests, and they are as effective (only less convenient).
Starting a new project today, it does not make any sense not to have Unit Tests, but I'd say that for an existing project like yours it does not make too much sense to write them for everything you already have and it's working.
In your case, I'd rather use a "middle ground" approach writing:
smaller Integration Tests which only test the sections you are going to refactor. If you are refactoring the whole thing, then you can use your current Integration Tests, but if you are refactoring only -say- the XML generation, it does not make any sense to require the presence of the database, so I'd write a simple and small XML Integration Test.
a bunch of Unit Tests for the new code you are going to write. As I already wrote above, Unit Tests will be ready as soon as you "mess with anything in between", making sure that your "mess" is going somewhere.
In fact your Integration Test will only make sure that your "mess" is not working (because at the beginning it will not work, right?) but it will not give you any clue on
why it is not working
if your debugging of the "mess" is really fixing something
if your debugging of the "mess" is breaking something else
Integration Tests will only give the confirmation at the end if the whole change was successful (and the answer will be "no" for a long time). The Integration Tests will not give you any help during the refactoring itself, which will make it harder and possibly frustrating. You need Unit Tests for that.
I agree with Charlie about Integration-level tests corresponding more to user actions and the correctness of the system as a whole. I do think there is alot more value to Unit Tests than just localizing failures more tightly though. Unit tests provide two main values over integration tests:
1) Writing unit tests is as much an act of design as testing. If you practice Test Driven Development/Behavior Driven Development the act of writing the unit tests helps you design exactly what you code should do. It helps you write higher quality code (since being loosely coupled helps with testing) and it helps you write just enough code to make your tests pass (since your tests are in effect your specification).
2) The second value of unit tests is that if they are properly written they are very very fast. If I make a change to a class in your project can I run all the corresponding tests to see if I broke anything? How do I know which tests to run? And how long will they take? I can guarantee it will be longer than well written unit tests. You should be able to run all of you unit tests in a couple of minutes at the most.
Just a few examples from personal experience:
Unit Tests:
(+) Keeps testing close to the relevant code
(+) Relatively easy to test all code paths
(+) Easy to see if someone inadvertently changes the behavior of a method
(-) Much harder to write for UI components than for non-GUI
Integration Tests:
(+) It's nice to have nuts and bolts in a project, but integration testing makes sure they fit each other
(-) Harder to localize source of errors
(-) Harder to tests all (or even all critical) code paths
Ideally both are necessary.
Examples:
Unit test: Make sure that input index >= 0 and < length of array. What happens when outside bounds? Should method throw exception or return null?
Integration test: What does the user see when a negative inventory value is input?
The second affects both the UI and the back end. Both sides could work perfectly, and you could still get the wrong answer, because the error condition between the two isn't well-defined.
The best part about Unit testing we've found is that it makes devs go from code->test->think to think->test->code. If a dev has to write the test first, [s]he tends to think more about what could go wrong up front.
To answer your last question, since unit tests live so close to the code and force the dev to think more up front, in practice we've found that we don't tend to refactor the code as much, so less code gets moved around - so tossing and writing new tests constantly doesn't appear to be an issue.
The question has a philisophical part for sure, but also points to pragmatic considerations.
Test driven design used as the means to become a better developer has its merits, but it is not required for that. Many a good programmer exists who never wrote a unit test. The best reason for unit tests is the power they give you when refactoring, especially when many people are changing the source at the same time. Spotting bugs on checkin is also a huge time-saver for a project (consider moving to a CI model and build on checkin instead of nightly). So if you write a unit test, either before or after you written the code it tests, you are sure at that moment about the new code you've written. It is what can happen to that code later that the unit test ensures against - and that can be significant. Unit tests can stop bugs before tehy get to QA, thereby speeding up your projects.
Integration tests stress the interfaces between elements in your stack, if done correctly. In my experience, integration is the most unpredictable part of a project. Getting individual pieces to work tends not to be that hard, but putting everything together can be very difficult because of the types of bugs that can emerge at this step. In many cases, projects are late because of what happens in integration. Some of the errors encountered in this step are found in interfaces that have been broken by some change made on one side that was not communicated to the other side. Another source of integration errors are in configurations discovered in dev but forgotten by the time the app goes to QA. Integration tests can help reduce both types dramatically.
The importance of each test type can be debated, but what will be of most importance to you is the application of either type to your particular situation. Is the app in question being developed by a small group of people or many different groups? Do you have one repository for everything, or many repos each for a particular component of the app? If you have the latter, then you will have challenges with inter compatability of different versions of different components.
Each test type is designed to expose the problems of different levels of integration in the development phase to save time. Unit tests drive the integration of the output many developers operating on one repository. Integration tests (poorly named) drive the integration of components in the stack - components often written by separate teams. The class of problems exposed by integration tests are typically more time-consuming to fix.
So pragmatically, it really boils down to where you most need speed in your own org/process.
The thing that distinguishes Unit tests and Integration tests is the number of parts required for the test to run.
Unit tests (theoretically) require very (or no) other parts to run.
Integration tests (theoretically) require lots (or all) other parts to run.
Integration tests test behaviour AND the infrastructure. Unit tests generally only test behaviour.
So, unit tests are good for testing some stuff, integration tests for other stuff.
So, why unit test?
For instance, it is very hard to test boundary conditions when integration testing. Example: a back end function expects a positive integer or 0, the front end does not allow entry of a negative integer, how do you ensure that the back end function behaves correctly when you pass a negative integer to it? Maybe the correct behaviour is to throw an exception. This is very hard to do with an integration test.
So, for this, you need a unit test (of the function).
Also, unit tests help eliminate problems found during integration tests. In your example above, there are a lot of points of failure for a single HTTP call:
the call from the HTTP client
the servlet validation
the call from the servlet to the business layer
the business layer validation
the database read (hibernate)
the data transformation by the business layer
the database write (hibernate)
the data transformation -> XML
the XSLT transformation -> HTML
the transmission of the HTML -> client
For your integration tests to work, you need ALL of these processes to work correctly. For a Unit test of the servlet validation, you need only one. The servlet validation (which can be independent of everything else). A problem in one layer becomes easier to track down.
You need both Unit tests AND integration tests.
Unit tests execute methods in a class to verify proper input/output without testing the class in the larger context of your application. You might use mocks to simulate dependent classes -- you're doing black box testing of the class as a stand alone entity. Unit tests should be runnable from a developer workstation without any external service or software requirements.
Integration tests will include other components of your application and third party software (your Oracle dev database, for example, or Selenium tests for a webapp). These tests might still be very fast and run as part of a continuous build, but because they inject additional dependencies they also risk injecting new bugs that cause problems for your code but are not caused by your code. Preferably, integration tests are also where you inject real/recorded data and assert that the application stack as a whole is behaving as expected given those inputs.
The question comes down to what kind of bugs you're looking to find and how quickly you hope to find them. Unit tests help to reduce the number of "simple" mistakes while integration tests help you ferret out architectural and integration issues, hopefully simulating the effects of Murphy's Law on your application as a whole.
Joel Spolsky has written very interesting article about unit-testing (it was dialog between Joel and some other guy).
The main idea was that unit tests is very good thing but only if you use them in "limited" quantity. Joel doesn't recommend to achive state when 100% of your code is under testcases.
The problem with unit tests is that when you want to change architecture of your application you'll have to change all corresponding unit tests. And it'll take very much time (maybe even more time than the refactoring itself). And after all that work only few tests will fail.
So, write tests only for code that really can make some troubles.
How I use unit tests: I don't like TDD so I first write code then I test it (using console or browser) just to be sure that this code do nessecary work. And only after that I add "tricky" tests - 50% of them fail after first testing.
It works and it doesn't take much time.
We have 4 different types of tests in our project:
Unit tests with mocking where necessary
DB tests that act similar to unit tests but touch db & clean up afterwards
Our logic is exposed through REST, so we have tests that do HTTP
Webapp tests using WatiN that actually use IE instance and go over major functionality
I like unit tests. They run really fast (100-1000x faster than #4 tests). They are type safe, so refactoring is quite easy (with good IDE).
Main problem is how much work is required to do them properly. You have to mock everything: Db access, network access, other components. You have to decorate unmockable classes, getting a zillion mostly useless classes. You have to use DI so that your components are not tightly coupled and therefore not testable (note that using DI is not actually a downside :)
I like tests #2. They do use the database and will report database errors, constraint violations and invalid columns. I think we get valuable testing using this.
#3 and especially #4 are more problematic. They require some subset of production environment on build server. You have to build, deploy and have the app running. You have to have a clean DB every time. But in the end, it pays off. Watin tests require constant work, but you also get constant testing. We run tests on every commit and it is very easy to see when we break something.
So, back to your question. Unit tests are fast (which is very important, build time should be less than, say, 10 minutes) and the are easy to refactor. Much easier than rewriting whole watin thing if your design changes. If you use a nice editor with good find usages command (e.g. IDEA or VS.NET + Resharper), you can always find where your code is being tested.
With REST/HTTP tests, you get a good a good validation that your system actually works. But tests are slow to run, so it is hard to have a complete validation at this level. I assume your methods accept multiple parametres or possibly XML input. To check each node in XML or each parameter, it would take tens or hundreds of calls. You can do that with unit tests, but you cannot do that with REST calls, when each can take a big fraction of a second.
Our unit tests check special boundary conditions far more often than #3 tests. They (#3) check that main functionality is working and that's it. This seems to work pretty well for us.
As many have mentioned, integration tests will tell you whether your system works, and unit tests will tell you where it doesn't. Strictly from a testing perspective, these two kinds of tests complement each other.
I can't see how adding a bunch more
unit tests on individual classes would
help. Wouldn't that make it harder to
refactor as it's much more likely we
will need to throw away and re-write
tests?
No. It will make refactoring easier and better, and make it clearer to see what refactorings are appropriate and relevant. This is why we say that TDD is about design, not about testing. It's quite common for me to write a test for one method and in figuring out how to express what that method's result should be to come up with a very simple implementation in terms of some other method of the class under test. That implementation frequently finds its way into the class under test. Simpler, more solid implementations, cleaner boundaries, smaller methods: TDD - unit tests, specifically - lead you in this direction, and integration tests do not. They're both important, both useful, but they serve different purposes.
Yes, you may find yourself modifying and deleting unit tests on occasion to accommodate refactorings; that's fine, but it's not hard. And having those unit tests - and going through the experience of writing them - gives you better insight into your code, and better design.
Although the setup you described sounds good, unit testing also offers something important. Unit testing offers fine levels of granularity. With loose coupling and dependency injection, you can pretty much test every important case. You can be sure that the units are robust; you can scrutinise individual methods with scores of inputs or interesting things that don't necessarily occur during your integration tests.
E.g. if you want to deterministically see how a class will handle some sort of failure that would require a tricky setup (e.g. network exception when retrieving something from a server) you can easily write your own test double network connection class, inject it and tell it to throw an exception whenever you feel like it. You can then make sure that the class under test gracefully handles the exception and carries on in a valid state.
You might be interested in this question and the related answers too. There you can find my addition to the answers that were already given here.
A Unit-Test should
produce deterministic result
be independent
be valid
...
What other characteristics should a test also have?
Ah. My favorite subject :-) Where to start...
According to xUnit test patterns by Gerard Meszaros (THE book to read about unit testing)
Tests should reduce risk, not
introduce it.
Tests should be easy to run.
Tests should be easy to maintain as
the system evolves around them
Some things to make this easier:
Tests should only fail because of one
reason. (Tests should only test one thing, avoid multiple asserts for example.)
There should only be one test that fails for that reason. (this keeps your testbase maintainable)
Minimize test dependencies (no
dependencies on databases, files, ui
etc.)
Other things to look at:
Naming
Have a descriptive name. Tests-names should read like specifications. If your names get too long you're probably testing too much.
Structure
Use AAA structure. This is the new fad for mocking frameworks, But I think it's a good way to structure all your tests like this.
Arrange your context
Act, do the things that need to be tested
Assert, assert what you want to check
I usually divide my tests in three blocks of code. Knowing this pattern makes tests more readable.
Mocks vs. Stubs
When using a mocking framework always try to use stubs and state based testing before resorting to mocking.
Stubs are objects that stand in for dependencies of the object you're trying to test. You can program behaviour into them and they can get called in your tests. Mocks expand on that by letting you assert if they were called and how. Mocking is very powerfull but it lets you test implementation instead of pre and post-conditions of your code. This tends to make tests more brittle.
The Pragmatic Programmers' answer : good tests shall be A-TRIP
Automatic
Thorough
Repeatable
Independent
Professional
not access external resources
be readable
Automatable: no manual intervention should be required to run the tests (CI).
Complete: they must cover as much code they can (Code Coverage).
Reusable: no need to create tests that will only be executed once.
Independent: Independent execution of a test should not affect the performance of another.
Professional: tests should be considered with the same value as the code, the same professionalism, documentation, etc.
One that I haven't seen anyone else mention is small. A unit test should test for one particular thing and that's all. I try to aim to have only one assert and minimize the amount of setup code by refactoring them out into their own methods. I'll also create my own custom asserts. A nice small unit test IMO is about 10 lines or less. When a test is small it is easy to get a quick understanding of what the test is trying to do. Large tests end up being unmaintainable in the long run.
Of course, small isn't the only thing I aim for...its just one of the things I value in a unit test. :-)
An other factors to keep in mind is the running time. If a test runs too long it is likely to be skipped.
Must be fully automatic.
Must not assume any preconditions
(product X be installed, file and Y
location etc).
Must be person independent as far as
running the scripts are concerned. Result can, however, be analysed by
subject experts only.
Must run on every beta build.
Must produce a verifiable report.
A unit test should be fast: hundreds of test should be able to run in a few seconds.
A test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing config files) to run it.
Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.
source: A Set of Unit Testing Rules