When writing unit tests that deal with XML (e.g. test a class that reads/generates XML) I used to write my asserted outcome XML-String / my input XML String in separate files right next to my unit test. Let's say I have a class "MyTransformer" that transformes one XML format into another. Then I would create three files all in the same package:
MyTransformerTest.java
MyTransformerTestSampleInput.xml
MyTransformerTestExpectedOutput.xml
Then my assertion might look like this (simplified pseudo code for reasons of simplicity):
Reader transformed = MyTransformer.transform(getResourceAsStream("MyTransformerTestSampleInput.xml")));
Reader expected = getResourceAsStream("MyTransformerTestExpectedOutput.xml");
assertXMLEqual(expected, transformed);
However a colleague told me that the file access that I have in this unit test is unacceptable. He proposed creating a literal string constant (private static final String) containing my XML file contents, possibly in a separate groovy class because of the benefit of multi line strings rather than writing the XML file into files.
I dislike the idea of the literal string constants, because even if I have multi line strings in groovy, I still loose syntax highlighting and all the other helpful features of my XML editor that tell me right away if my XML has syntax errors etc.
What do you think? Is the file access really bad? If so: Why? If not why is it ok?
Two problems with files in unit tests:
they slow down the testing cycle. You may have thousands of unit tests which, preferably, get run on every build - so they should be as fast as possible. If you can speed them up (eg, by getting rid of I/O operations) you'd want to do that. Surely it's not always feasible, so you normally separate out the "slow" tests via NUnit [Category] or something similar - and then run those special tests less frequently - say, only on Nightly builds.
they introduce additional dependencies. If a test requires a file, it will fail not only when the logic behind the test is wrong, but also when the file is missing, or test runner doesn't have read permissions etc. Which makes debugging and fixing not so pleasing!
That said, I won't be too strict about not using files in the tests. If possible, try to avoid them but don't get mad. Make sure you consider maintainability vs speed - the cleaner the test, the easier it will be to fix and understand it later.
If your Unit Tests access the files to feed the fake test data into the System Under Test, so you can run tests, that's not a problem. That actually helps your to have wider variety of test data to exercise within the system under test.
However if your System Under Test access the file system, when executing from the a test, that's not a Unit Test. That's an Integration Test. This is because you are accessing cross cutting concerns such as file system, and they cannot categorised as Unit Tests.
You would really isolate/fake out the file access and test the behaviour of your code (if any), using Unit Tests. They are faster and easier to run. It gives your a pin point feedback if written correctly.
In these cases, I have a unit test that uses an internal representation of the file, which is a string literal in this case.
I also will have an Integration Test, to test the code works correctly when writing to the file.
So it is all down to the Unit / Integration test definitions. Both are valid tests, just depends which test you are writing at the time.
If the xml is more readable, or easier to work with in a file, and you have a lot of these tests, I would leave them.
Strictly speaking, unit tests should not use the file system because it is slow. However, readability is more important. XML in a file is easier to read, and can be loaded in an XML friendly editor.
If the tests take to long to run (cause you have a lot of them), or your colleagues complain, move them to integration tests.
If you work in Windows and LINUX, you have to be careful that the files are picked up by your build server.
There are no perfect answers.
Related
Is it a code smell to use a generated file, like a spreadsheet or XML file, as a source for comparison in Unit tests?
Say that we had to write a lot of classes that produce various XML files for processing later. The unit tests would be a large number of repetitive assertions that foo.getExpectedValue() = expectedValue. Instead of doing this, the developer chooses to use the code they are supposed to be testing to generate the XML, then copy that into test/resources for all future tests load into memory as an object and to run their assertions against. Is this a code smell?
There are two practices in what you describe that classify as test smells.
First, you write that the classes that are to be tested are used to create the XML files that are later used to judge the correctness. This way you can find out if there are changes in the classes, but you can not figure out if the results were correct in the first place.
To avoid any misunderstandings: The smell is not that generated files are used, but that the files are generated with the code under test. The only way how such an approach might make sense would be if the results of the initial run were subject to thorough review. But, these reviews would have to be repeated again whenever the files are re-generated later.
Secondly, using complete XML files for comparison (generated or not) is another test small. The reason is, that these tests are not very focused. Any change to the XML generation will lead to a failing test. That may seem like a good thing, but it even applies to all kinds of intended changes, for example to changes in the indentation. Thus, you only have test that tell you "something changed", but not "something failed".
To have tests that tell you "something failed" you need more specific tests. For example, tests that only look at a certain portion of the generated XML. Some tests would look at the XML structure, others at the data content. You can even have tests to check the indentation. But, how? You could, for example, use regular expressions to see if some interesting portion of a generated XML string looks the way you expected.
Once you have more focused tests, then the rest results in case of intended modifications to your code will look different: When your changes are successfull, only a few test cases of many will fail, and this will be the ones that have tested against the part of the behaviour that you intentionally have changed. All other tests will still work OK and show you that your change did not break something unexpectedly. If in contrast your change was incorrect, then some more/other tests than the expected ones will show you that the change had unexpected effects.
Yes, I wouldn't do that. This is breaking principles of a good test. Mainly, a good test test should be;
Independent - should not rely on other tests. If subsequent tests have to wait for a file generated by the first test,tests are not independent.
Repeatable - these tests introduce the flakiness due to the file/in-memory dependency. Therefore, they may not be repeatable consistently.
Perhaps, you could take a step back and see whether you need to unit test every generated XML. If these file generation follow the same code execution path with (no logical difference), I wouldn't write unit test per each case. If these XML generation is business operation related, I would consider having acceptance tests.
If the files are small, you can use string reader and writer. In general one is supposed to avoid doing i/o in unit tests. However,I see nothing wrong with using files in some non-unit tests.
I have an architectural dilemma regarding JUnit Test structure.
The flow of my test class test the loading of a dictionary from a large file. The class may fail due to lack of memory, file not found, wrong file structure, and so on.
On the first hand, it makes good sense to run the tests on a specific order: check for memory, then check if the file exists, then check for its structure. This would be very helpful, because if we don't have enough memory, the appropriate test would fail and give meaningful output, instead of the cryptic OutOfMemoryException. Moreover, I won't have to overgo the time-consuming process of reading the file over and over again.
On the other hand, tests should be isolated, self contained entities.
Any ideas how to desing the test suite?
What you are describing is an integration test, rather than a unit test. Unit tests (ideally) should not rely on the file system, amount of memory available etc.
You should set up your unit tests with a manageable amount of data in the dictionary (preferably initialized e.g. from a string (stream) within the test fixture code, not from an external file). This way you can reinitialize your test fixture independently for each unit test case, and test whatever nitty gritty details you need to about the structure of the dictionary.
And of course, you should also create separate, high level integration tests to verify that your system as a whole works as expected under real life circumstances. If these are separated from the genuine unit tests, you can afford to run the (cheap and fast) unit tests via JUnit after any code change, and run the costlier integration tests via some other way only more rarely, depending on how long they take, e.g. once every night. You may also choose between setting up the integration environment once, and running your integration tests (via some other means than JUnit, e.g. a shell / batch script) in a well defined order, or reloading the dictionary file for each test case, as the extended execution time may not matter during a nightly build.
To add to what Péter says:
An OOME unit test wouldn't rely on actually running out of memory. Instead whatever calls for the file load should react appropriately when the code-that-loads throws the OOME. This is what mocking/etc. are for--simulating behavior outside of what is being tested. (Same for missing files/file structure errors.)
IMO loading a file that's too big for the JVM configuration isn't really a useful thing to test in the code that actually loads the file--as long as whatever calls for the file load handles an OOME appropriately--at least not in a unit test.
If the code-that-loads needs some behavior verification when the file is too big I'd consider stubbing out the file load part and force an OOME rather than actually load a giant file, just to save time. To reiterate his point, actual physical behavior can be moved to integration tests and, say, run on a CI server outside of your own TDD heartbeat.
I've written an xml parser, and I've written a unit test to go with it.
Where should I store the actual xml file to be used by the test?
Should I just store it under test/unit/whatever package the test is in?
It depends on several things:
is this going to be a single test, or are you planning to develop more tests (and test files)? In the former case, it does not matter so much, while in the latter case you might want to think about orgnaizing your test data files in advance. In any case I would store the test file(s) in a separate test directory structure (which may nevertheless mimic the package structure).
is this a kind of "smoketest" to be published with the parser (this is what your description suggests to me), or only for internal use? In the former case you might prefer organizing your test files in a user-friendly manner; users typically are not concerned so much with the internal package structure of your app, but with ease of use and logical organization. This would dictate organizing (and naming!) the test files according to the use cases they are related.
Update: So, multiple unit test files for internal use. I would then just put them into a test directory, making it easy and uniform for the tests to find them. If they are in the same package where the test class is, whenever you happen to move a class to a different package, you must remember to move the xml file too, and update the file path inside the test. Otherwise the test will happily continue running with the test file in the old directory, without you noticing anything! This opens up possibilities for subtle bugs over time - like deleting a test file which seems to be unused, or worse: overwriting it with another test file...
A Unit-Test should
produce deterministic result
be independent
be valid
...
What other characteristics should a test also have?
Ah. My favorite subject :-) Where to start...
According to xUnit test patterns by Gerard Meszaros (THE book to read about unit testing)
Tests should reduce risk, not
introduce it.
Tests should be easy to run.
Tests should be easy to maintain as
the system evolves around them
Some things to make this easier:
Tests should only fail because of one
reason. (Tests should only test one thing, avoid multiple asserts for example.)
There should only be one test that fails for that reason. (this keeps your testbase maintainable)
Minimize test dependencies (no
dependencies on databases, files, ui
etc.)
Other things to look at:
Naming
Have a descriptive name. Tests-names should read like specifications. If your names get too long you're probably testing too much.
Structure
Use AAA structure. This is the new fad for mocking frameworks, But I think it's a good way to structure all your tests like this.
Arrange your context
Act, do the things that need to be tested
Assert, assert what you want to check
I usually divide my tests in three blocks of code. Knowing this pattern makes tests more readable.
Mocks vs. Stubs
When using a mocking framework always try to use stubs and state based testing before resorting to mocking.
Stubs are objects that stand in for dependencies of the object you're trying to test. You can program behaviour into them and they can get called in your tests. Mocks expand on that by letting you assert if they were called and how. Mocking is very powerfull but it lets you test implementation instead of pre and post-conditions of your code. This tends to make tests more brittle.
The Pragmatic Programmers' answer : good tests shall be A-TRIP
Automatic
Thorough
Repeatable
Independent
Professional
not access external resources
be readable
Automatable: no manual intervention should be required to run the tests (CI).
Complete: they must cover as much code they can (Code Coverage).
Reusable: no need to create tests that will only be executed once.
Independent: Independent execution of a test should not affect the performance of another.
Professional: tests should be considered with the same value as the code, the same professionalism, documentation, etc.
One that I haven't seen anyone else mention is small. A unit test should test for one particular thing and that's all. I try to aim to have only one assert and minimize the amount of setup code by refactoring them out into their own methods. I'll also create my own custom asserts. A nice small unit test IMO is about 10 lines or less. When a test is small it is easy to get a quick understanding of what the test is trying to do. Large tests end up being unmaintainable in the long run.
Of course, small isn't the only thing I aim for...its just one of the things I value in a unit test. :-)
An other factors to keep in mind is the running time. If a test runs too long it is likely to be skipped.
Must be fully automatic.
Must not assume any preconditions
(product X be installed, file and Y
location etc).
Must be person independent as far as
running the scripts are concerned. Result can, however, be analysed by
subject experts only.
Must run on every beta build.
Must produce a verifiable report.
A unit test should be fast: hundreds of test should be able to run in a few seconds.
A test is not a unit test if:
It talks to the database
It communicates across the network
It touches the file system
It can't run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing config files) to run it.
Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.
source: A Set of Unit Testing Rules
I'm looking to unit testing as a means of regression testing on a project.
However, my issue is that the project is basically a glorified DIR command -- it performs regular expression tests and MD5 filters on the results, and allows many criteria to be specified, but the entire thing is designed to process input from the system on which it runs.
I'm also a one-man development team, and I question the value of a test for code written by me which is written by me.
Is unit testing worthwhile in this situation? If so, how might such tests be accomplished?
EDIT: MD5 and Regex functions aren't provided by me -- they are provided by the Crypto++ library and Boost, respectively. Therefore I don't gain much by testing them. Most of the code I have simply feeds data into the libraries, and the prints out the results.
The value of test-after, the way you are asking, can indeed be limited in certain circumstances, but the way to unit test, from the description would be to isolate the regular expression tests and MD5 filters into one section of code, and abstract the feeding of the input so that in production it feeds from the system, and during the unit test, your test class passes in that input.
You then collect a sampling of the different scenarios you intend to support, and feed those in via different unit tests that exercise each scenario.
I think the value of the unit test will come through if you have to change the code to handle new scenarios. You will be confident that the old scenarios don't break as you make changes.
Is unit testing worthwhile in this situation?
Not necessarily: especially for a one-man team I think it may be sufficient to have automated testing of something larger than a "unit" ... further details at "Should one test internal implementation, or only test public behaviour?"
Unit testing can still provide value in a one-man show. It gives you confidence in the functionality and correctness (at some level) of the module. But some design considerations may be needed to help make testing more applicable to your code. Modularization makes a big difference, especially if combined with some kind of dependency injection, instead of tight coupling. This allows test versions of collaborators to be used for testing a module in isolation. In your case, a mock file system object could return a predictable set of data, so your filtering and criteria code can be evaluated.
The value of regression testing is often not realized until it's automated. Once that's done, things become a lot easier.
That means you have to be able to start from a known position (if you're generating MD5s on files, you have to start with the same files each time). Then get one successful run where you can save the output - that's the baseline.
From that point on, regression testing is simply a push-button job. Start your test, collect the output and compare it to your known baseline (of course, if the output ever changes, you'll need to check it manually, or with another independent script before saving it as the new baseline).
Keep in mind the idea of regression testing is to catch any bugs introduced by new code (i.e., regressing the software). It's not to test the functionality of that new code.
The more you can automate this, the better, even as a one-man development team.
When you were writing the code, did you test it as you went? Those tests could be written into an automated script, so that when you have to make a change in 2 months time, you can re-run all those tests to ensure you haven't regressed something.
In my experience the chance of regression increases sharply depending on how much time goes by after the time you finish version 1 and start coding version 2, because you'll typically forget the subtle nuances of how it works under different conditions - your unit tests are a way of encoding those nuances.
An integration test against the filesystem would be worthwhile. Just make sure it does what it needs to do.
Is unit testing valuable in a one-man shop scenario? Absolutely! There's nothing better than being able to refactor your code with absolute confidence you haven't broken anything.
How do I unit test this? Mock the system calls, and then test the rest of your logic.
I question the value of a test for code written by me which is written by me
Well, that's true now but in a year it will be you, the one-year-more-experienced developer developing against software written by you-now, the less experienced and knowledgeable developer (by comparison). Won't you want the code written by that less experienced guy (you a year ago) to be properly tested so you can make changes with confidence that nothing has broken?