Exactly how independent should unit tests be? What should be done in a "before" section of a unit testing suite?
Say, for example, I am testing the functionality of a server - should the server be created, initialised, connected to it's various data sources, &c. inside the body of every test case. Are there situations where it may be appropriate to initialise the server once, and then test more than one case.
The other situation I am considering is mobile app testing - where the phone objects need to be created to perform a unit test. Should this be done every time. Create phone, initialise, run test, destroy phone, repeat?
Unit tests should be completely independent, i.e. each should be able to run in any order so each will need to have its own initialization steps.
Now, if you are talking about server or phone initialization, it sounds more like integration tests rather than unit tests.
Ideally yes. Every test should start from scratch, and put the system into a particular well-defined state before executing the function under test. If you don't, then you make it more difficult to isolate the problem when the test fails. Even worse, you may cause the test not to fail because of some extra state left behind by an earlier test.
If you have a situation where the setup time is too long, you can mock or stub some of the ancillary objects.
If you are worried about having too much setup code, you can refactor the setup code into reusable functions.
Related
I have a large program that I need to write tests for. I'm wondering if it would be wrong to write tests that run in a specific order, as some necessarily have to run in order and that depend upon a previous test.
For example a scenario like the following:
CreateEmployer
CreateEmployee (requires employer)
Add Department
The drawback I see to this approach is that if one test fails, all the of the follow tests will also fail. But, I am going to have to write the code to build the database anyway, so it might be a more effective approach to use the code that builds the mock database as a sort of integration-like unit test.
Should I create the database without using the tests as a seed method, and then run each of the methods again to see the result? The problem I see with this approach is that if the seed method does not work all of the tests will fail and it won't be immediately clear that the error is in the seed method and not the services or the tests themselves.
Yes, this is discouraged. Tests shouldn't be "temporally coupled".
Each test should run in complete isolation of other tests. If you find yourself in a situation where the artifacts created by Test A are needed by Test B then you have two problems to correct:
Test A shouldn't be creating artifacts (or side-effects).
Test B should be using mock data as part of its "arrange" step.
Basically, the unit tests shouldn't be using a real database. They should be testing the logic of the code, not the interaction with the infrastructure dependencies. Mock those dependencies in order to test just the code.
It is a bad idea to have unit tests depend upon each other. In your example you could have a defect in both CreateEmployer and AddDepartment, but when all three fail because of the CreateEmployer test you might mistakenly assume that only the CreateEmployer test is 'really' failing. This means you've lost the potently valuable information that AddDepartment is failing as well.
Another problem is that you might create a separate workflow in the future that calls AddDepartment without calling CreateEmployer. Now your tests assume that CreateEmployer will always be called, but in reality it isn't. All three tests could pass, but the application could still break because you have a dependency you didn't know was there.
The best tests won't rely on a database at all, but will instead allow you to manually specify or "Mock" the data. Then you don't have to worry about an unrelated database problem breaking all of your tests.
If these are truly Unit tests then yes, requiring a specific order is a bad practice for several reasons.
Coupling - As you point out, if one test fails, then subsequent ones will fail as well. This will mask real problems.
TDD - A core principle of TDD is make tests easy to run. If you do so, developers are more likely to run them. If they are hard to run (e.g. I have to run the entire suite), then they are less likely to be run and their value is lost.
Ideally, unit tests should not depend upon the completion of another test in order to run. It is also hard to run them in a given sequence, but that may depend upon your unit testing tool.
In your example, I would create a test that tests the CreateEmployer method and makes sure it returns a new object the way you expect.
The second test I would create would be CreateEmployee, and if that test requires an Employer object, using dependency injection, your CreateEmployee method could receive its Employer object. Here is where you would use a mock object (one that code to get created, returning a fixed/known Employer) as the Employer object the CreateEmployee method would consume. This lets you test the CreateEmployee method and its actions upon that object, with a given/known instance of the Employer object.
Your third test, AddDepartment, I assume also depends upon an Employer object. This unit test can follow the same pattern, and receive a mock Employer object to consumer during its test. (The same object you pass to unit test two above.)
Each test now runs/fails on its own, and can run in any sequence.
Basically I have two main questions:
What exactly should you unit test?
How do you do it?
The problem is I have several applications that rely in a database connection and/or are communication applications, which mean most of the test cases are integration tests (or so I think).
Most classes are fairly simple by themselves, but the ones that implement the communication protocol, which are the ones that would be useful to automate the testing, can seem to fit well into the "unit test" model.
Another example. I developed I pipe structure with multithreaded support for a consumer/producer pattern. When a thread reads the pipe and finds it empty it blocks until a writer writes into the pipe. Should I use unit tests to test that class?
How do you decide what to unit test?
Edit: I mean writing unit tests for automated unit testing.
You Unit tests units of your code. The real question is what exactly makes up a unit?
In an object oriented environment a unit is a class. A class because behaviours of an object vary with the state of the object, so testing a method in isolation will not yeild the most complete results.
First you need to identify the invariants of the class. That is, the things that will always be true for all instances ofthe class. E.g. in a Fraction class an invariant may be denominator != 0.
Next you need to identify the contracts of each method, that is, the pre and post conditions of the methods.
Then you write tests for each condition that may arise. So for a single class you may end up with many test methods to test the various conditions that each method could encounter. At each test you ensure that the invariants of the class holds and the contract of the method is never broken.
In some cases like the example that you provided it may be necessary to create other objects in the environment in order to test the conditions of your class. In those instances you may use mock objects.
You should abstract your infrastructure concerns (ie code that retrieves data from your database, code that does file i/o, etc) so that you can stub/mock those parts in order to unit test your application. And then you will be able to write targeted/specific tests against your infrastructure code to test that out.
You will be finding yourself creating more interfaces (to create seems within your application), and needing to use better OO principles (ie. SOLID) in order to develop an application that is more testable.
I was in the same boat a year ago that you were in. And the one book that really helped me through it (along with some hands on practice) is The Art of Unit Testing by Roy Osherove
Unit tests test units (that is method or function) in isolation, in a dedicated, controlled environment. Each unit test creates in own environment by instantiating only the classes needed to execute one test, putting them in a known state, then it invokes the method to be tested and verify the outcome. This verification is done by assertions on the behavior of the method (as opposed to its implementation).
Performing the verification on the behavior and not on the implementation is important as this allows modifying the implementation without breaking the unit tests, and therefore using the unit tests as a safety net for the modification.
All language have [at least] one unit test framework whose role is to execute the unit tests. There are two ways to write unit tests: test-first or test-last.
Test-first is also called Test-Driven Development. Basically it takes three steps:
write a failing test
write just enought code to make it pass
refactor the code to clean it up (remove duplication ...)
Proponents of TDD claim that this leads to testable code, while it could be hard to write unit tests after the fact, especially when methods do several things. It is recommended to follow the Single Responsibility Principle.
Regarding the pipe structure and communications protocol example, some guidelines say that
a test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
...
When a thread reads the pipe and finds
it empty it blocks until a writer
writes into the pipe. Should I use
unit tests to test that class?
I would test the class, but not the blocking read method, as I presume it is build from a blocking call to the read() function of the Operating System.
Unit testing is testing the entire unit works the same as it did before the change, with the improvements made in the change.. If you are making a change to a window of a module, then you test the module. This is in comparison to a system test, which is to test every module. If your change affects many modules (not sure how your system is set up), then it should get a system test.
Here's another way to think about it - particularly if you are not focusing on object oriented code, but perhaps more functional or procedural code.
Ideally, unit tests should cover every path through the code. It should be your chance to see every path through your code works as expected and as needed. If you are practicing Test Driven Development then it's implied everything gets a test.
I think looking at this blog post will help clarify: https://earnestengineer.blogspot.com/2018/03/unit-testing-what-to-test.html
I'm just getting stated with boost-test and unit testing in general with a new application, and I am not sure how to handle the applications initialisation (eg loading config files, connecting to a database, starting an embedded python interpretor, etc).
I want to test this initialisation process, and also most of the other modules in the application require that the initialisation occurred successfully.
Some way to run some shut down code would also be appreciated.
How should I go about doing this?
It seems what you intent to do is more integration test than unit-test. It's not to pinpoint on wording, but it makes a difference. Unit testing mean testing methods in isolation, in an environment called a fixture, created just for one test, end then deleted. Another instance of the fixture will be re-created if the next case require the same fixture. This is done to isolate the tests so that an error in one test does not affect the outcome of the subsequent tests.
Usually, one test has three steps:
Arrange - prepare the fixture : instantiate the class to be tested, possibly other objects needed
Act - call the method to be tested
Assert - verify the expectations
Unit tests typically stays away of external resources such as files and databases. Instead mock objects are used to satisfy the dependencies of the class to be tested.
However, depending on the type of your application, you can try to run tests from the application itself. This is not "pure" unit testing, but can be valuable anyway, especially if the code has not been written with unit testing in mind, it might not be "flexible" enough to be unit tested.
This need a special execution mode, with a "-test" parameter for instance, which will initialize the application normally, and then invoke tests that will simulate inputs and use assertions to verify the application reacted as expected. Likewise, it might be possible to invoke the shutdown code and verify with assertions if the database connection has be closed (if the objects are not deleted).
This approach has several drawbacks compared to unit tests: it depends on the config files (the software may behave differently depending on the parameters), on the database (on its content and on the ability to connect to it), the tests are not isolated ... The two first can be overcome using default values for the configuration and connecting to a test database in test mode.
Are you defining BOOST_TEST_MAIN? If so, and you have no main function of your own (where you would otherwise put initialization code) you could feasibly use some form of singleton object that exposes an init function that you can call before each test if required.
I am looking for rules like:
A test is not a unit-test if:
it communicates with a database
it cannot run in parallel with other tests
uses the "environment" like registry or file system
What else is there?
See Michael Feathers' definition
A test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing
config files) to run it.
A test is not a unit test if it is not testing a unit.
Seriously, that's all there is to it.
The concept of "unit" in unit testing is not well-defined, in fact, the best definition I have found so far, isn't actually a definition because it is circular: a unit in a unit test is the smallest possible thing that can be tested in isolation.
This gives you two checkpoints: is it tested in isolation? And is it the smallest possible thing?
Please note that both of these are context-dependent. What might be the smallest possible thing in one situation (say, an entire object) might in another situation just one small piece of one single method. And what counts as isolation in one situation might be in another (e.g. in a memory-managed language, you never run in isolation from the garbage collector, and most of the time that is irrelevant, but sometimes it might not be).
Difficult one...
For me a unit test verifies one specific piece of logic in isolation. Meaning, I take some logic, extract it from the rest (if necessary by mocking dependencies) and test just that logic - a unit (of the whole) - by exploring different kind of possible control flows.
But on the other side...can we always 100% say correct or incorrect?? Not to become philosophical, but - as also Michael says in his post:
Tests that do these things aren't bad.
Often they are worth writing, and they
can be written in a unit test harness.
However, it is important to be able to
separate them from true unit tests so
that we can keep a set of tests that
we can run fast whenever we make our
changes.
So why shouldn't I write a unit test that verifies the logic of parsing for instance an xls file by accessing some dummy file from the file system in my test folder (like MS tests allow with the DeploymentItem)?
Of course - as mentioned - we should separate these kind of tests from the others (maybe in a separate test suite in JUnit). But I think one should also write those tests if he feels comfortable in having them there...clearly then always again remembering that a unit test should just test a piece in isolation.
What is most important in my eyes is that these tests run fast and don't take too long s.t. they can be run repeatedly and very often.
It has no asserts, and is not expecting an exception to be thrown.
A test is not an Unit Test when:
it tests more than one thing at once (i.e. it tests how two things work together) - then it is an integration test
Checklist for good unit tests:
they are automated
they are repeatable
they are easy to implement
they remain for future use, once written
they can be run by anyone
they can be run by the push of a button
they run quickly
Some more best practices (in no particular order of importance):
tests should be separated from integration tests (which are slower), so that they can be run fast as frequently as possible
they should not comprise too much logic (preferably, no control structures)
every test should test only one thing (thus, they should contain only one assert)
the expected values used in assertions should be hard-coded and not computed at test run-time
external dependencies (filesystem, time, memory etc.) should be replaced by stubs
test should recreate the initial state at test shutdown
in assertions, it is better to use a "contains..." policy, rather than "is strictly equal..." policy (i.e. we expect certain values in a collection, certain characters in a string etc.)
This is part of the knowledge I have extracted from Roy Osherove's book - The Art of Unit Testing
Implementing a test across multiple possibly failing units would not be a unit test.
Intricate question.
Say I am to program some business logic and all business logic needs to get to the data via some form of DAL.
Say that for the purposes of testing, I mock the DAL units (by creating "mockingbirds").
But those mockingbirds are of course, additional units in their own right. So even when using mocks, it might seem like I'm still bound to violate the idea of "no other units involved" when I want to unit-test my business logic module.
Of course, it is generally known that "creating mockingbirds for the DAL" can invalidate your very test itself on the count that your mockingbird deviates in some particular aspect from the DAL.
Conclusion : it is outright impossible to do "genuine unit-tests" on business modules that depend in any way on any kind of DAL, question mark ?
Corrolary : the only thing that can possible be ("genuinely" !) unit-tested is the DAL itself, question mark ?
Corrolary of the corrolary : given that the "DAL" is usually either an ORM or the very DML of some DBMS, and given that those products are usually bought as being "proven technology", what is the added value of doing any unit tests what so ever, question mark ?
After whether a test is a unit test or not is settled the next question is, is it a good unit test?
I have been creating Unit tests like crazy and find that I'm often having to set up something in one test that I just tore down in a previous test. Is it ever reasonable to create something (e.g. a database record) in one test (e.g. an Insertion test) and then use it for a later test (e.g. a Deletion test)? Or should each and every test always stand completely on its own?
Can you even determine the order of tests in NUnit or are they always done alphabetically?
Note: I am specifically asking about the order of tests within one test file. Not across test files or in any way more globally.
Update: Thanks to everyone that answered - there were a lot of good answers and the sense of the group is pretty unanimous. I've chosen John Nolan's answer as he provided the most complete explanation and lots of links. As you may have guessed, I've been sorely tempted to break this rule despite thinking that it might be a bit "smelly" as John put it. Thanks also to Fortyrunner for adding the unit-testing tag.
Look into test fixture setups that allow you to specify functions that will be executed before any of the tests in the fixture. This allows you to do common setup once and it will always run, whether you run one test, or all tests in the suite.
Relying on the order of your tests indicates that you are persisting state across tests. This is smelly
A cleaner way of testing is where you only depend on the single piece of functionality you want to check the behaviour of. Commonly you mock the other objects you need to get your method under test to function.
A good way to think about approaching unit tests is the Arrange, Act, Assert pattern.
Below is a snippet from Karl Seguin's excellent free eBook. I've annoted Arrange, Act and Assert.
[TestFixture] public class CarTest
{
[Test] public void SaveCarCallsUpdateWhenAlreadyExistingCar()
{
//Arrange
MockRepository mocks = new MockRepository();
IDataAccess dataAccess = mocks.CreateMock<IDataAccess>();
ObjectFactory.InjectStub(typeof(IDataAccess), dataAccess);
//Act
Car car = new Car();
Expect.Call(dataAccess.Save(car)).Return(389);
mocks.ReplayAll();
car.Save();
mocks.VerifyAll();
// Assert
Assert.AreEqual(389, car.Id);
ObjectFactory.ResetDefaults();
}
}
Unit tests are intended to stand alone, not be run as a sequential script. If you actually need them run sequentially, collect them into a single test function.
If your unit tests suffer from expensive set-up, you may be doing integration testing when you think you're doing unit testing. If you're hitting a SQL database inside most of your unit tests, you're actually integration testing with your data access layer.
I would view each test as completely independent from any other test. Even if you could mandate the order of the tests, it would be a maintenance nightmare when the tests must change.
I really wouldn't rely on ordering of tests. Instead, I'd pull the common setup code into a separate method and call that from both the simple test and the more complicated one. Alternatively, just call the insertion test itself at the start of the deletion test.
I would strongly advise to make all your unit tests independent.
Your business logic / database structure etc. may change over time, so that you'll eventually have to replace or rewrite (or even discard) existing unit tests - and if you have several other tests depending on the one that you're replacing, this might cause unnecessary troubles because you'd have to go through all of the other tests as well and check if these are still working as expected.
In addition, one failing unit test should not be able to drag many others (that might perfectly work on their own) down.
Unfortunately run order of the unit test is not predictable or at least could be changed in future. E.g. unit testing framework will be changed so each test will be executed in separate thread.
So from my point of view using test order is not reasonable.
On the other hand you can create a set of small independent tests to test small parts of your code and then create one or several large tests that will run your small tests in specific order.
If you have stateful tests (a common problem with database work - which is what I do when I'm not on SO), then it seems to me that avoiding order within a test file is not absolutely necessary. However, you have to recognize that if you have 2 tests, with test 2 depending on test 1 passing, then you will get a 'catastrophic' double failure if test 1 fails, because test 2 doesn't have the expected setup (and, what's more, you want to worry if test 2 does pass after test 1 failed if you think that test 2 depends on test 1 passing).
That's why you want the tests independent whenever possible - both intra-file and inter-file.
It would be very unwise to depend on the order between (sets of) tests in different files.