Is there a clean way to run parameterized jUnit 4 tests without changing the runner, i.e. without using
#RunWith(Parameterized.class)?
I have unit tests which require a special runner already and I can't replace this one with Parameterized. Maybe there is some kind of "runner chaining" so I could both runners at the same time? (Just a wild guess...)
I have released a framework with a couple of runners that are able to enforce parameterization on the test-class while allowing you to chain an arbitrary 3rd-party runner for the actual test-execution.
The framework is CallbackParams - (http://callbackparams.org) - and these are the runners:
CallbackParamsRunner
BddRunner
By using the framework annotation ...
#WrappedRunner
... you can specify an arbitrary 3rd-party runner in this manner:
#RunWith(CallbackParamsRunner.class) // or #RunWith(BddRunner.class)
#WrappedRunner(YourSpecialRunner.class)
public class YourTest {
...
Parameterized tests with CallbackParams differ considerably from the traditional approach to test-parameterization, however. The reasons are explained in this tutorial article with BddRunner explained near the end of the tutorial article.
For your first CallbackParams test you would probably prefer BddRunner, since it requires less boiler-plate stuff, but when you start reusing parameter values between different test-classes you are probably better off with CallbackParamsRunner, which demands stronger type-checking.
Also - with BddRunner you must not have any #Test-methods. Instead you must use the framework annotations #Given, #When and #Then. That requirement sometimes clash with those of the third-party runner but it usually works out quite well.
Good Luck!
org.junit.runners.Parameterized is created by org.junit.internal.builders.AnnotatedBuilder by reflect mechanism. Maybe you could extend Parameterized as your own Runner: #RunWith(MyParameterized.class).
Related
I'm just curious, are there any standard guidelines that state whether an instance of a class under test should be constructed in a fixture or in the actual test case?
Thanks!
I'm not aware of a standard reference on that topic. Here's what I'd do:
If I had only one test to write, or if I needed an instance of the class under test that was constructed differently than any other instance of that class in my test suite, I'd just instantiate it in the test. Why make it any more complicated that you have to? If I needed to use the same instance over and over again, I'd put it in a fixture.
I do think it's important to construct only the fixtures you need for a given test case, so that there's nothing to mislead the reader. That means either using whatever scoping mechanism your test framework provides (e.g. an rspec context block or a whole new xUnit TestCase) to construct a given fixture only before the tests that need it, or moving instance construction from fixtures to test. To avoid duplication, you can always write a method to construct an instance and call it from as many tests as you want.
I tend to avoid putting anything inside a fixture.
After a while the CUT state tends to get out of hand as the number of tests in that fixture increase. Each test require a simialr but different behavior which could or could not be added to some initialization/setup method.
Having the CUT in the fixture level creates a shared states between the tests which can cause test failure due to run order - which is a pain to find and fix.
Another readability issue happens when a test fails - people tend to forget the initialization that might have happened in another method.
There are better ways to avoid code duplication - Using AutoMocking container to create objects with fake parameters or Factory methods which enable a different initialization for each test (if required) and create more readable and maintainable tests.
I'm having trouble getting my unit tests to stay independent of each other. For instance, I have a linked list with two append methods, one that takes a single element and appends it to the list, and one that takes another list and appends the whole thing; but I can't test the second append method (the one that takes a whole list) without using the first append method to populate the list I'm passing in. How do I keep the unit tests for these two methods separate from each other?
The situation you describe happens everywhere in testing: You have some class or library to test. The class or library has certain methods / functions that need to be tested, and for the test of some of these, you have to call other methods / functions of that same library.
In other words, when breaking down the test according to the four phase test pattern (setup, exercise, evaluate, cleanup), you want to call your class / lib in the exercise phase. However, it seems annoying that you have to call some elements of it also in the setup phase, and, possibly, also in the evaluate and/or cleanup phases.
This is simply unavoidable: You mentioned that in the setup for the list append function you had to use the single-element append function. But, it is even worse: You also had to use the constructor of your list class - no chance to get away without that one. But, the constructor could also be buggy...
What can certainly happen is, that tests fail (or, mistakenly pass) because the functions called in the setup are defective. A proper test suite, however, should (as was mentioned in the comments) also have tests for the other (call them lower-level) functions.
For example, you should have a number of tests which check that the constructor of your class works correctly. If at some point you modify the constructor so it becomes defective, all tests that use the constructor in the setup phase are no longer trustworthy. But, some of the tests that test the constructor itself (and thus call it in the exercise phase) should fail now.
From the overview of the test results you will then be able to identify the root cause of the test failures. This requires some understanding about the dependencies: Which of the tests focus on the lower-level aspects and which are higher-level in the sense that they depend on some lower-level functionality to work.
There are some ways to make these dependencies more apparent and therefore make it easier to analyse test failures later - but none of these are essential:
In the test-code, you put the tests for the lower-level aspects at the top of the file, and the more dependent tests further to the bottom. Then, when several tests fail, first look at the test that is closest to the top of the file. Note that the order of tests in the test code does not necessarily imply an execution order: JUnit for example does not care in which order the test methods are written in the test class.
As it was suggested in the comments, you may in addition configure the test framework to run the lower level tests before the others.
You can create one method which itself is not a unit test method but instead creates the conditions for multiple tests, then performs verification of the results. Your actual unit test methods will call into this other method. So you can use the same data set for multiple tests, and not introduce dependencies between test methods.
I don't know what language you are using, but here is an example for Objective-C in Xcode 5 with the new XCTest framework. I would do something like this:
- (void)performTestWithArray:(NSArray *)list
{
NSMutableArray *initialList = ...; // create the initial list you will use with multiple tests
[initialList addObjectsFromArray:list];
XCTAssertTrue(testCondition, #"message");
}
- (void)testAddSingleElement
{
NSArray *array = #[ #"one element" ];
[self performTestWithArray:array];
}
- (void)testAddList
{
NSArray *array = #[ #"first element", #"second element", #"third element" ];
[self performTestWithArray:array];
}
I have just started to read Professional Test Driven Development with C#: Developing Real World Applications with TDD
I have a hard time understanding stubs, fakes and mocks. From what I understand so far, they are fake objects used for the purpose of unit testing your projects, and that a mock is a stub with conditional logic into it.
Another thing I think I have picked up is that mocks are somehow related with dependency injection, a concept which I only managed to understand yesterday.
What I do not get is why I would actually use them. I cannot seem to find any concrete examples online that explains them properly.
Can anyone please explain to me this concepts?
As I've read in the past, here's what I believe each term stands for
Stub
Here you are stubbing the result of a method to a known value, just to let the code run without issues. For example, let's say you had the following:
public int CalculateDiskSize(string networkShareName)
{
// This method does things on a network drive.
}
You don't care what the return value of this method is, it's not relevant. Plus it could cause an exception when executed if the network drive is not available. So you stub the result in order to avoid potential execution issues with the method.
So you end up doing something like:
sut.WhenCalled(() => sut.CalculateDiskSize()).Returns(10);
Fake
With a fake you are returning fake data, or creating a fake instance of an object. A classic example are repository classes. Take this method:
public int CalculateTotalSalary(IList<Employee> employees) { }
Normally the above method would be passed a collection of employees that were read from a database. However in your unit tests you don't want to access a database. So you create a fake employees list:
IList<Employee> fakeEmployees = new List<Employee>();
You can then add items to fakeEmployees and assert the expected results, in this case the total salary.
Mocks
When using mock objects you intend to verify some behaviour, or data, on those mock objects. Example:
You want to verify that a specific method was executed during a test run, here's a generic example using Moq mocking framework:
public void Test()
{
// Arrange.
var mock = new Mock<ISomething>();
mock.Expect(m => m.MethodToCheckIfCalled()).Verifiable();
var sut = new ThingToTest();
// Act.
sut.DoSomething(mock.Object);
// Assert
mock.Verify(m => m.MethodToCheckIfCalled());
}
Hopefully the above helps clarify things a bit.
EDIT:
Roy Osherove is a well-known advocate of Test Driven Development, and he has some excellent information on the topic. You may find it very useful :
http://artofunittesting.com/
They are all variations of the Test Double. Here is a very good reference that explains the differences between them: http://xunitpatterns.com/Test%20Double.html
Also, from Martin Fowler's post: http://martinfowler.com/articles/mocksArentStubs.html
Meszaros uses the term Test Double as the generic term for any kind of
pretend object used in place of a real object for testing purposes.
The name comes from the notion of a Stunt Double in movies. (One of
his aims was to avoid using any name that was already widely used.)
Meszaros then defined four particular kinds of double:
Dummy objects: are passed around but never actually used. Usually they
are just used to fill parameter lists.
Fake objects actually have working implementations, but usually take some shortcut which makes
them not suitable for production (an in memory database is a good
example).
Stubs provide canned answers to calls made during the test,
usually not responding at all to anything outside what's programmed in
for the test. Stubs may also record information about calls, such as
an email gateway stub that remembers the messages it 'sent', or maybe
only how many messages it 'sent'.
Mocks are what we are talking about here: objects pre-programmed with expectations which form a
specification of the calls they are expected to receive.
Of these kinds of doubles, only mocks insist upon behavior verification. The
other doubles can, and usually do, use state verification. Mocks
actually do behave like other doubles during the exercise phase, as
they need to make the SUT believe it's talking with its real
collaborators.
This PHP Unit's manual helped me a lot as introduction:
"Sometimes it is just plain hard to test the system under test (SUT) because it depends on other components that cannot be used in the test environment. This could be because they aren't available, they will not return the results needed for the test or because executing them would have undesirable side effects. In other cases, our test strategy requires us to have more control or visibility of the internal behavior of the SUT." More: https://phpunit.de/manual/current/en/test-doubles.html
And i find better "introductions" when looking for "test doubles" as mocks, fakes, stubs and the others are known.
I have a PersonDao that I'm writing unit tests against.
There are about 18-20 methods in PersonDao of the form -
getAllPersons()
getAllPersonsByCategory()
getAllPersonsUnder21() etc
My Approach to testing this was to create a PersonDaoTest with about 18 test methods testing each of the method in PersonDao
Then I created a PersonDaoPaginationTest that tested these 18 methods by applying pagination parameters.
Is this in anyway against the TDD best practices? I was told that this creates confusion and is against the best practices since this is non-standard. What was suggested is merging the two classes into PersonDaoTest instead.
As I understand is, the more broken down into many classes your code is, the better, please comment.
The fact that you have a set of 18 tests that you are going to have to duplicate to test a new feature is a smell that suggests that your PersonDao class is taking on multiple responsibilities. Specifically, it appears to be responsible both for querying/filter and for pagination. You may want to take a look at whether you can do a bit of design work to extract the pagination functionality into a separate class which could then be tested independently.
But in answer to your question, if you find that you have a class that you want to remain complex, then it's perfectly fine to use multiple test classes as a way of organizing a large number of tests. #Gishu's answer of grouping tests by their setup is a good approach. #Ryan's answer of grouping by "facets" or features is another good approach.
Can't give you a sweeping answer without looking at the code... except use whatever seems coherent to you and your team.
I've found that grouping tests based on their setup works out nicely in most cases. i.e if 5 tests require the same setup, they usually fit nicely into a test-fixture. if the 6th test requires a different setup (more or less) break it out into a separate test fixture.
This also leads to test-fixtures that are feature-cohesive (i.e. tests grouped on feature), give it a try. I'm not aware of any best practice that says you need to have one test class per production class... in practice I find I have n test classes per production classes, the best practice would be to use good names and keep related tests close (in a named folder).
My 2 cents: when you have a large class like that that has different "facets" to it, like pagination, I find it can often make for more understandable tests to not pack them all into one class. I can't claim to be a TDD guru, but I practice test-first development religiously, so to speak. I don't do it often, but it's not exactly rare, either, that I'll write more than a single test class for a particular class. Many people seem to forget good coding practices like separation of concerns when writing tests, though. I'm not sure why.
I think one test class per class is fine - if your implementation has many methods, then your test class will have many methods - big deal.
You may consider a couple of things however:
Your methods seem a bit "overly specific" and could use some abstraction or generalisation, for example instead of getAllPersonsUnder21() consider getAllPersonsUnder(int age)
If there are some more general aspects of your class, consider testing them using some common test code using call backs. For a trivial example to illustrate testing that both getAllPersons() returns multiple hits, do this:
#Test
public void testGetAllPersons() {
assertMultipleHits(new Callable<List<?>> () {
public List<?> call() throws Exception {
return myClass.getAllPersons(); // Your call back is here
}
});
}
public static void assertMultipleHits(Callable<List<?>> methodWrapper) throws Exception {
assertTrue("failure to get multiple items", methodWrapper.call().size() > 0);
}
This static method can be used by any class to test if "some method" returns multiple hits. You could extends this to do lots of tests over the same callback, for example running it with and without a DB connection up, testing that it behaves correctly in each case.
I'm working on test automation of a web app using selenium. It is not unit testing but you might find that some principles apply. Tests are very complex and we figured out that the only way to implement tests in a way that meets all our requirements was having 1 test per class. So we consider that each class is an individual test, then, we were able to use methods as the different steps of the test. For example:
public SignUpTest()
{
public SignUpTest(Map<String,Object> data){}
public void step_openSignUpPage(){}
public void step_fillForm(){}
public void step_submitForm(){}
public void step_verifySignUpWasSuccessfull(){}
}
All the steps are dependent, they follow the order specified and if someone fail the others will not be executed.
Of course, each step is a test by itself, but they all together form the sing up test.
The requirements were something like:
Tests must be data driven, this is, execute the same test in parallel with different inputs.
Tests must run in different browsers in parallel as well. So each
test will run "input_size x browsers_count" times in parallel.
Tests will focus in a web workflow, for example, "sign up with valid data" and they will be split into smaller tests units for each step of the workflow. It will make things easier to
maintain, and debug (when you have a failure, it will say:
SignUpTest.step_fillForm() and you'll know immediately what's wrong).
Tests steps share the same test input and state (for example, the id of the user created). Imagine if you put in the same class
steps of different tests, for example:
public SignUpTest()
{
public void signUpTest_step_openSignUpPage(){}
public void signUpTest_step_step_fillForm(){}
public void signUpTest_step_step_submitForm(){}
public void signUpTest_step_verifySignUpWasSuccessfull(){}
public void signUpNegativeTest_step_openSignUpPage(){}
public void signUpNegativeTest_step_step_fillFormWithInvalidData(){}
public void signUpNegativeTest_step_step_submitForm(){}
public void signUpNegativeTest_step_verifySignUpWasNotSuccessfull(){}
}
Then, having in the same class state belonging to the 2 tests will be
a mess.
I hope I was clear and you may find this useful. At the end, choosing what will represent your test: if a class or a method is just a decision that I think will depend int: what is the target of a test (in my case, a workflow around a feature), what's easier to implement and maintain, if a test fail how you make the failure more accurate and how you make it easier to debug, what will lead you to more readable code, etc.
I think understand the definition of State / Interaction based testing (read the Fowler thing, etc). I found that I started state based but have been doing more interaction based and I'm getting a bit confused on how to test certain things.
I have a controller in MVC and an action calls a service to deny a package:
public ActionResult Deny(int id)
{
service.DenyPackage(id);
return RedirectToAction("List");
}
This seems clear to me. Provide a mock service, verify it was called correctly, done.
Now, I have an action for a view that lets the user associate a certificate with a package:
public ActionResult Upload(int id)
{
var package = packageRepository.GetPackage(id);
var certificates = certificateRepository.GetAllCertificates();
var view = new PackageUploadViewModel(package, certificates);
return View(view);
}
This one I'm a bit stumped on. I'm doing Spec style tests (possibly incorrectly) so to test this method I have a class and then two tests: verify the package repository was called, verify the certificate repository was called. I actually want a third to test to verify that the constructor was called but have no idea how to do that! I'm get the impression this is completely wrong.
So for state based testing I would pass in the id and then test the ActionResult's view. Okay, that makes sense. But wouldn't I have a test on the PackageUploadViewModel constructor? So if I have a test on the constructor, then part of me would just want to verify that I call the constructor and that the action return matches what the constructor returns.
Now, another option I can think of is I have a PackageUploadViewModelBuilder (or something equally dumbly named) that has dependency on the two repositories and then I just pass the id to a CreateViewModel method or something. I could then mock this object, verify everything, and be happy. But ... well ... it seems extravagant. I'm making something simple ... not simple. Plus, controller.action(id) returning builder.create(id) seems like adding a layer for no reason (the controller is responsible for building view models.. right?)
I dunno... I'm thinking more state based testing is necessary, but I'm afraid if I start testing return values then if Method A can get called in 8 different contexts I'm going to have a test explosion with a lot of repetition. I had been using interaction based testing to pass some of those contexts to Method B so that all I have to do is verify Method A called Method B and I have Method B tested so Method A can just trust that those contexts are handled. So interaction based testing is building this hierarchy of tests but state based testing is going to flatten it out some.
I have no idea if that made any sense.
Wow, this is long ...
I think Roy Osherove recently twitted that as a rule of thumb, your tests should be 95 percent state-based and 5 percent interaction-based. I agree.
What matters most is that your API does what you want it to, and that is what you need to test. If you test the mechanics of how it achieves what it needs to do, you are very likely to end up with Overspecified Tests, which will bite you when it comes to maintainability.
In most cases, you can design your API so that state-based testing is the natural choice, because that is just so much easier.
To examine your Upload example: Does it matter that GetPackage and GetAllCertificates was called? Is that really the expected outcome of the Upload method?
I would guess not. My guess is that the purpose of the Upload method - it's very reason for existing - is to populate and serve the correct View.
So state-based testing would examine the returned ViewResult and its ViewModel and verify that it has all the correct values.
Sure, as the code stands right now, you will need to provide Test Doubles for packageRepository and certificateRepository, because otherwise exceptions will be thrown, but it doesn't look like it is important in itself that the repository methods are being called.
If you use Stubs instead of Mocks for your repositories, your tests are no longer tied to internal implementation details. If you later on decide to change the implementation of the Upload method to use cached instances of packages (or whatever), the Stub will not be called, but that's okay because it's not important anyway - what is important is that the returned View contains the expected data.
This is much more preferrable than having the test break even if all the returned data is as it should be.
Interestingly, your Deny example looks like a prime example where interaction-based testing is still warranted, because it is only by examining Indirect Outputs that you can verify that the method performed the correct action (the DenyPackage method returns void).
All this, and more, is explained very well in the excellent book xUnit Test Patterns.
The question to ask is "if this code worked, how could I tell?" That might mean testing some interactions or some state, it depends on what's important.
In your first test, the Deny changes the world outside the target class. It requires a collaboration from a service, so testing an interaction makes sense. In your second test, you're making queries on the neighbours (not changing anything outside the target class), so stubbing them makes more sense.
That's why we have a heuristic of "Stub Queries, Mock Actions" in http://www.mockobjects.com/book