I have a C++ legacy codebase with 10-15 applications, all sharing several components.
While setting up unittests for both shared components and for applications themselves, I was wondering if there are accepted/common file structures for this.
Because my unit tests have several base classes in order to simplify project/customer specific test setups, there are alot of files that are common for all tests.
To me it seems natural here to create a new directory that contains all the test related files, mocks etc -to have it all centralized, and also keep testing related definitions out of the main make files.
On the other hand I see that it is common practice to have the test files reside together with the code files that they test.
Is there a more/less accepted way of doing this?
Out of sight, out of mind; if you keep the test files together with the code files it may be more obvious to the developers that when they update a code file they should update the tests as well.
As you noted, there are two common ways to locate unit test files: near the implementation code they are testing, and in a separate file hierarchy. The choice is a matter of what is the common practice in your organisation and personal taste.
Regarding the location of common test code, just organize your test code are you would organize implementation code.
In your particular case, if some test infrastructure is common to several independent components, it would be a good idea to create a new component (call it "testing", for example) that other components depends on for their tests, instead of adding dependences between existing components.
I usually organize such code in a file structure that looks (in a simple case) like this:
apps
app1
app1module1
app2module2
app1tests
app2
app2module1
app2tests
components
comp1
comp1module1
comp1module2
comp1tests
common_test_stuff
There is no single right way to do this, but this seems to be a common practice that keeps production and test code separate and attempts to remove the out-of-sight, out-of-mind problem (mentioned by zac) at the same time.
Keep the test code close to the product code, and arrange your Makefile (or whatever you're using) so that the tests compile at the same time as the test, to make them visible, especially if not everyone in the team is writing tests.
Related
I am using Visual Studio 2017 Unit Test Projects.
I have an application that reads complex XML files, and do some processing based on XPATH queries in their data. I need to build a series of unit tests for my application. Below two options come to my mind:
Option #1: Include a few selected XML files as part of the Unit Test Project.
Create unit tests that reads the files, and tests specific scenarios.
This is the most intuitive and easiest option for me. Because I can include many complex scenarios in my selected XML files update them as required, then build all the required unit tests.
Option #2: Create mock classes that provide specific XML data scenarios
I need to create a few mock classes that provide specific XML data scenarios. The application needs to be updated such that it gets the data through classes in an IoC container.
To me, Option#1 is quite simple, but I wanted to get more information on the Option#2
Question:
Are above two options aligned with Unit Test design best practices? What is the most optimal Unit Test design for my senatio?
What is best depends a lot on your specific needs. However, typically you would strive for tests that run fast, execute in different environments (not only your machine), are deterministic etc.
To me it seems that option #1, while probably being the fastest to implement, has negative impact on test execution (file access takes some time), makes the test execution dependent on the environment (permissions to the file, which directory?, file name encoding, ...), can lead to non-deterministic behaviour (file being locked due to a parallel test process running) etc.
In contrast, option #2 will likely cost you more implementation effort initially. There are, however, mechanisms to make it less costly: You can implement helper functions to create typical XML content, such that several test cases can benefit from these common helper functions etc.
Even if we have a Makefile or something similar to separate the test code when shipping product.
In my opinion, they should be separate, but i am not entirely convinced as to WHY
Yes, they should be separate (folders and preferably projects). Some reasons:
GREP. Searching for a string in production source is easier.
Code coverage. Imagine trying to specify which files to include for coverage.
Different standards. You may want to run static analysis, etc. only on production code.
Simplified makefiles/build scripts.
Modern IDEs will allow you to work on code from separate projects/folders as if they were adjacent.
The worst thing you can do is to include test and production code in the same file (with conditional compilation, different entry points, etc.). Not only can this confuse developers trying to read the code, you always run the risk of accidentally shipping test code.
Since I had a chance to work with both approaches (separated and with project code), here's few tiny-things that were getting in a way to note (C#, Visual Studio, MsBuild).
Same project approach
References/external libraries dependencies: unit testing and mocking frameworks usually come with few dependencies on it's own, combine that with libraries you need for actual project and list grows very quickly (and nobody likes huge lists, right?)
Naming collisions: having class named MyClass, common approach is to name test class MyClassTest - this causes tiny annoyances when using navigation/naming completition tools (since there's bigger chance you'll have more than one result to chose from for quick navigation)
overall feeling of ubiquitous mess
Naming collisions can actually get even more tiresome, considering how classes relating to similar functionality usually share prefix (eg. ListManager ... Converter, Formatter, Provider). Navigating between comprehensible number of items (usually 3-7) is not a problem - enter tests, enter long lists again.
Separated approach
Projects number: you'll have to count the number of libraries you produce twice. Once for project code alone, another time for tests. When bigger projects are involved (200-300+ sub-projects/libraries) having that number doubled by test projects extends IDE startup time in a way you never want to experience
Of course, modern machines will mitigate projects number issue. Unless this really becomes a problem, I'd always go for separated projects approach - it's just more neat and clean and way easier to manage.
In fully automated building and unit testing framework, you can essentially separate them out.
It makes more sense to fire up the automated unit tests after the nightly build is done.
Keeping them separate makes it easier for maintenance purposes.
I have found several conventions to housekeeping unit tests in a project and
I'm not sure which approach would be suitable for our next PHP project. I am
trying to find the best convention to encourage easy development and
accessibility of the tests when reviewing the source code. I would be very
interested in your experience/opinion regarding each:
One folder for productive code, another for unit tests: This separates
unit tests from the logic files of the project. This separation of
concerns is as much a nuisance as it is an advantage: Someone looking into
the source code of the project will - so I suppose - either browse the
implementation or the unit tests (or more commonly: the implementation
only). The advantage of unit tests being another viewpoint to your classes
is lost - those two viewpoints are just too far apart IMO.
Annotated test methods: Any modern unit testing framework I know allows
developers to create dedicated test methods, annotating them (#test) and
embedding them in the project code. The big drawback I see here is that
the project files get cluttered. Even if these methods are separated
using a comment header (like UNIT TESTS below this line) it just bloats
the class unnecessarily.
Test files within the same folders as the implementation files: Our file
naming convention dictates that PHP files containing classes (one class
per file) should end with .class.php. I could imagine that putting unit
tests regarding a class file into another one ending on .test.php would
render the tests much more present to other developers without tainting
the class. Although it bloats the project folders, instead of the
implementation files, this is my favorite so far, but I have my doubts: I
would think others have come up with this already, and discarded this
option for some reason (i.e. I have not seen a java project with the files
Foo.java and FooTest.java within the same folder.) Maybe it's because
java developers make heavier use of IDEs that allow them easier access to
the tests, whereas in PHP no big editors have emerged (like eclipse for
java) - many devs I know use vim/emacs or similar editors with little
support for PHP development per se.
What is your experience with any of these unit test placements? Do you have
another convention I haven't listed here? Or am I just overrating unit test
accessibility to reviewers?
I favour keeping unit-tests in separate source files in the same directory as production code (#3).
Unit tests are not second-class citizens, their code must maintained and refactored just like production code. If you keep your unit tests in a separate directory, the next developer to change your production code may miss that there are unit tests for it and fail to maintain the tests.
In C++, I tend to have three files per class:
MyClass.h
MyClass.cpp
t_MyClass.cpp
If you're using Vim, then my toggle_unit_tests plug-in for toggling between source and unit test files may prove useful.
The current best practice is to separate the unit tests into their own directory, #1. All of the "convention over configuration" systems do it like this, eg. Maven, Rails, etc.
I think your alternatives are interesting and valid, and the tool support is certainly there to support them. But it's just not that popular (as far as I know). Some people object to having tests interspersed with production code. But it makes sense to me that if you always write unit tests, that they be located right with your code. It just seems simpler.
I always go for #1. While it's nice that they are close together. My reasons are as followed:
I feel there's a difference between the core codebase and the unittests. I need a real separation.
End-users rarely need to look at unittests. They are just interested in the API. While it's nice that unittests provide a separate view on the code, in practice I feel it won't be used to understand it better. (more descriptive documentation + examples do).
Because end-users rarely need unittests, I don't want to confuse them with more files and/or methods.
My coding standards are not half as strict for unittests as they are for the core library. This is maybe just my opinion, but I don't care as much for coding standards in my tests.
Hope this helps.
I've written an xml parser, and I've written a unit test to go with it.
Where should I store the actual xml file to be used by the test?
Should I just store it under test/unit/whatever package the test is in?
It depends on several things:
is this going to be a single test, or are you planning to develop more tests (and test files)? In the former case, it does not matter so much, while in the latter case you might want to think about orgnaizing your test data files in advance. In any case I would store the test file(s) in a separate test directory structure (which may nevertheless mimic the package structure).
is this a kind of "smoketest" to be published with the parser (this is what your description suggests to me), or only for internal use? In the former case you might prefer organizing your test files in a user-friendly manner; users typically are not concerned so much with the internal package structure of your app, but with ease of use and logical organization. This would dictate organizing (and naming!) the test files according to the use cases they are related.
Update: So, multiple unit test files for internal use. I would then just put them into a test directory, making it easy and uniform for the tests to find them. If they are in the same package where the test class is, whenever you happen to move a class to a different package, you must remember to move the xml file too, and update the file path inside the test. Otherwise the test will happily continue running with the test file in the old directory, without you noticing anything! This opens up possibilities for subtle bugs over time - like deleting a test file which seems to be unused, or worse: overwriting it with another test file...
I do TDD, and I've been fairly loose in organizing my unit tests. I tend to start with a file representing the next story or chunk of functionality and write all the unit-tests to make that work.
Of course, if I'm introducing a new class, I usually make a separate unit-test module or file for that class, but I don't organize the tests themselves into any higher level structure. The result is I write code fast and I believe my actual program is reasonably well structured, but the unit tests themselves are "messy". Especially, their structure tends to recapitulate the phylogeny of the development process. Sometimes I see myself as trading laziness in the code for laziness in the tests.
How big a problem is this? Who here continually refactors and reorganizes their unit tests to try to improve their overall structure? Any tips for this? What should the overall structure of tests look like.
(Note, that I'm not so much asking the "how many assertions per function" question asked here : How many unit tests should I write per function/method? I'm talking about the bigger picture.)
Divide your tests in 2 sets:
functional tests
units tests
Functional tests are per-user story. Unit tests are per-class. The former check that you actually support the story, the latter exercise and document your functionality.
There is one directory (package) for functional tests. Unit tests should be closely bound with functionality they exercise (so they're scattered). You move them around and refactor them as you move & refactor your code around.
The less important part is organizing the tests.
I start by putting the tests into a class that relates to the class under test, so com.jeffreyfredrick.Foo has a test com.jeffreyfredrick.FooTest. But if some subset of those classes need a different setup then I'll move them into their own test class. I put my tests into a separate source directory but keep them in the same project.
The more important part is refactoring the tests.
Yes I try and refactor my tests as I go. The goal is to remove duplication while still remaining declarative and easy to read. This is true both within test classes and across test classes. Within a test class I might have a parametrized method for creating a test fake (mock or stub). My test fakes are usually inner classes within a test class but if I find there's need I'll pull them out for reuse across tests. I'll also create a TestUtil class with common methods when it seems appropriate.
I think refactoring yours tests is important to long term success of unit testing on large projects. Have you ever heard people complaining about how their tests are too brittle or preventing them from change? You don't want to be in a position where changing the behavior of a class means making dozens or even hundreds of changes to your tests. And just like with code, you achieve this through refactoring and keeping the tests clean.
Tests are code.
I write a unit test class for each class in the application, and keep the test classes organized in the same package structure as the classes under test.
Inside each test class I don't really have much organizational structure. Each one only has a handful of methods for each public method in the class under test, so I've never had any problem finding what I'm looking for.
For every class in the software, I maintain a unit test class. The unit test classes follow the same package hierarchy as the classes which are tested.
I keep my unit test code in a separate project. Some people also prefer to keep their test code in the same project under a separate source directory called 'test'. You could follow whatever feels comfortable to you.
I try to look at the unit tests as a project on their own. As with any project the organisation should follow some internal logic. It does not however have to be specific or formally defined - anything you're comfortable with is OK as long as it keeps your project well-organised and clean.
So for the unit tests I usually either follow the main project code structure or (sometimes when the situation calls of it) focus on the functional areas instead.
Leaving them in one heap is as you might imagine messy and difficult to maintain