I know that this is subjective, but I'd like to follow the most common practice.
Do you normally create one test method for each class method and stuff it with multiple assertions, or do you create one test method per assertion?
For example, if I am testing a bank account's withdraw method, and I want make sure that an exception is thrown if the user tries to overdraw the account or withdraw a negative amount, should I create testOverdaw and testNegativeWithdrawal, or would I just combine those two assertions in a method called testWithdraw?
Think of it this way: each test should stand on its own and exercise a relatively discrete set of functionality. If you want to assert whether three things are true about some method that you have created, then you should create a test that includes those three things.
Thus, I have to strongly disagree with the others who have answered. Arbitrarily limiting yourself to one assertion per test will do nothing for you except make your testing unwieldy and tedious. Ultimately it may put you off testing altogether - which would certainly be a shame: bad for your project and career.
Now, that does not mean you have license to write large, unwieldy or multi-purpose testing routines. Indeed, I don't think I've ever written one that is more than 20 lines or so.
As far as knowing which assertion fails when there are several in one function, you will note that both nUnit and MSTest give you both a description and a link when an assertion fails that will take you right to the offending line (nUnit will require an integration tool such as TestDriven.net). This makes figuring out the failure point trivial. Both will also stop on the first failure in a function and both give you the ability to do a debug walkthrough as well.
Personally I would create one test for each assertion otherwise you have to dig to find the reason for the failure rather than it being obvious from the test name.
If you have to write a few lines of code to set up a test and don't want to duplicate that then depending on your language and test framework you should be able to create a set of tests where each test will execute a block of code before it runs.
Make multiple test methods; do not combine them into one. Each of your test methods should test one behavior of the method. As you are saying, testing with a negative balance is a different behavior then testing with a positive balance. So, that would be two tests.
You want to do it this way so that when a test fails, you are not stuck trying to figure out which part in that test failed.
One way to do it is have one separate method for each different scenario or setup. In your case you'd probably want one scenario where there is sufficient funds, and one scenario where there is insufficient funds. And assert that in the first one everything works, and in the second one the same operations won't work.
I would recommend having one test method per assertion, or rather per a expected behavior. This allows to localize the erroneous code much faster, if any test fails.
I would make those two seperate assertions.
The first represents a valid operation that would happen if a user was using the account regularly, the second would represent a case where data sanitizing was not done, or not properly done.
You want separate test cases so that you can logically implement the test cases as needed, especially in regression scenarios where running all tests can be prohibitively expensive.
testOverdraw and testNegativeWithdrawal are two separate behaviors. They shall be tested separately.
A good rule of thumb is to have only one action on the method under test in one unit test (not counting setup and assertions).
From the NUnit documentation:
"If an assertion fails, the method call does not return and an error is reported. If a test contains multiple assertions, any that follow the one that failed will not be executed. For this reason, it's usually best to try for one assertion per test."
http://www.nunit.org/index.php?p=assertions&r=2.4.6
However, nothing forces you to follow best practices. If it isn't worth your time and effort for this particular project to have 100% granular tests, where one bug means exactly one test failure (and vice-versa), then don't do it. Ultimately it is a tool for you to use at the best cost/benefit balance that you can. That balance depends on a lot of variables that are specific to your scenario, and specific to each project.
Related
The question might seem a bit weird, but i'll explain it.
Consider following:
We have a service FirstNameValidator, which i created for other developers so they have a consistent way to validate a person's first name. I want to test it, but because the full set of possible inputs is infinite (or very very big), i only test few cases:
Assert.IsTrue(FirstNameValidator.Validate("John"))
Assert.IsFalse(FirstNameValidator.Validate("$$123"))
I also have LastNameValidator, which is 99% identical, and i wrote a test for it too:
Assert.IsTrue(LastNameValidator.Validate("Doe"))
Assert.IsFalse(LastNameValidator.Validate("__%%"))
But later a new structure appeared - PersonName, which consists of first name and last name. We want to validate it too, so i create a PersonNameValidator. Obviously, for reusability i just call FirstNameValidator and LastNameValidator. Everything is fine till i want to write a test for it.
What should i test?
The fact that FirstNameValidator.Validate was actually called with correct argument?
Or i need to create few cases and test them?
That is actually the question - should we test what service is expected to do? It is expected to validate PersonName, how it does it we actually don't care. So we pass few valid and invalid inputs and expect corresponding return values.
Or, maybe, what it actually does? Like it actually just calls other validators, so test that (.net mocking framework allows it).
Unit tests should be acceptance criteria for a properly functioning unit of code...
they should test what the code should and shouldn't do, you will often find corner cases when you are writing tests.
If you refactor code, you often will have to refactor tests... This should be viewed as part of the original effort, and should bring glee to your soul as you have made the product and process an improvement of such magnitude.
of course if this is a library with outside (or internal, depending on company culture) consumers, you have documentation to consider before you are completely done.
edit: also those tests are pretty weak, you should have a definition of what is legal in each, and actually test inclusion and exclusion of at least all of the classes of glyphps... they can still use related code for testing... ie isValidUsername(name,allowsSpace) could work for both first name and whole name depending on if spaces are allowed.
You have formulated your question a bit strangely: Both options that you describe would test that the function behaves as it should - but in each case on a different level of granularity. In one case you would test the behaviour based on the API that is available to a user of the function. Whether and how the function implements its functionality with the help of other functions/components is not relevant. In the second case you test the behaviour in isolation, including the way the function interacts with its dependended-on components.
On a general level it is not possible to say which is better - depending on the circumstances each option may be the best. In general, isolating a piece of software requires usually more effort to implement the tests and makes the tests more fragile against implementation changes. That means, going for isolation should only be done in situations where there are good reasons for it. Before getting to your specific case, I will describe some situations where isolation is recommendable.
With the original depended-on component (DOC), you may not be able to test everything you want. Assume your code does error handling for the case the DOC returns an error code. But, if the DOC can not easily be made to return an error code, you have difficulty to test your error handling. In this case, if you double the DOC you could make the double return an error code, and thus also test your error handling.
The DOC may have non-deterministic or system-specific behaviour. Some examples are random number generators or date and time functions. If this makes testing your functions difficult, it would be an argument to replace the DOC with some double, so you have control over what is provided to your functions.
The DOC may require a very complex setup. Imagine a complex data base or some complex xml document that needs to be provided. For one thing, this can make your setup quite complicated, for another, your tests get fragile and will likely break if the DOC changes (think that the XML schema changes...).
The setup of the DOC or the calls to the DOC are very time consuming (imagine reading data from a network connection, computing the next chess move, solving the TSP, ...). Or, the use of the DOC prolongs compilation or linking significantly. With a double you can possibly shorten the execution or build time significantly, which is the more interesting the more often you are executing / building the tests.
You may not have a working version of the DOC - possibly the DOC is still under development and is not yet available. Then, with doubles you can start testing nevertheless.
The DOC may be immature, such that with the version you have your tests are instable. In such a case it is likely that you lose trust in your test results and start ignoring failing tests.
The DOC itself may have other dependencies which have some of the problems described above.
These criteria can help to come to an informed decision about whether isolation is necessary. Considering your specific example: The way you have described the situation I get the impression that none of the above criteria is fulfilled. Which for me would lead to the conclusion that I would not isolate the function PersonNameValidator from its DOCs FirstNameValidator and LastNameValidator.
A big chunk of our codebase has no unit tests whatsoever. We've been writing unit tests for any new code we add, but we're only just now starting to go back and add unit tests for the existing code. While writing & running unit tests for an existing calculation method, I found a bug: there's a particular input edge case that the calculation does not handle correctly.
The bug has never been detected before because it's not actually reachable in the application; the particular input that hits the bug is a subset of a group of inputs that are trivial and handled directly rather than being sent to the somewhat expensive calculation method. My boss has decided that since the bug can't be hit from the application, it's not worth digging through the calculation method to fix it.
Using XUnit, how should I best mark this bug as something we're aware of but have chosen not to fix? A failed test would break our build automation, so I can't just leave it as is. The input that fails is currently being generated as part of a PropertyData for a Theory.
Does XUnit have a special indicator for this? Should I adjust the method that generates the inputs for the PropertyData to exclude that case, add a comment explaining why, and then put in a skipped Fact covering that case?
You shouldn't have unit tests providing input data that your requirements state are not supported cases. In this case you don't have a bug, you simply have requirements stating that the given input is not supported and is considered invalid.
If you really want to, you can have tests that provide invalid input and assert failure, if you choose to make it an explicit requirement that this input must fail. If you don't want to do that, simply don't create tests for use cases you don't have.
Skipping the fact balances between not running the test but leaving noticeable warning that this is something to take care of in the future.
Alternatively, you can categorize the test and config the runner to skip it, see, e.g., the Category Sample v1,v2
I'm not used yet to write unit tests and I want to do this on a full framework of little tools (making it more secure to use). That way I'll certainly learn more about unit tests than what I learnt until now.
However I'm really used to add assertions systematically everywhere I see there is a context to be sure about (that are removed in the final release). Mostly as preconditions in functions implementations and each time I retrieve informations that have to be correct (like C/C++ pointers validity for a famous example).
Now I'm asking : are assertions redundant when you have unit tests? Because it looks redundant as you're testing the behaviour of a bit of code; but in the same time it's not the same execution context.
Should I do both?
Assertions that check preconditions can help detect and locate integration bugs. That is, whereas unit tests demonstrate that a method operates correctly when it is used (called) correctly, assertions that check preconditions can detect incorrect uses (calls) to the method. Using assertions causes faulty code to fail fast, which assists debugging.
Assertions not only validate the code but serve as a form of documentation that informs readers about properties of various structures that they can be sure will be satisfied at that point in execution (e.g. node->next != NULL). This helps to create a mental model of the code when you're reading through it.
Assertions also serve to prevent disaster scenarios during runtime. e.g.
assert(fuel);
launch_rocket();
Trying to launch when there is no fuel might be disastrous. Your unit tests might have caught this scenario but it's always possible that you've missed it and an abort because a condition is unmet is Way better than a crash and burn during runtime.
So, in short, I'd keep them there. It's a good habit to add them and there's no gain in unlearning it.
I would say that you need both. Unit tests test that your in-method asserts are correct ;)
Unit testing and assertions are vastly different things, yet they complement each other.
Assertions deals with proper input-output of methods.
Unit tests deal with ensuring a unit works according to some governing rules, e.g. a specification. A unit may be ax method, a class, or a set of collaborating classes (sometimes such tests go by the name "integration tests")
So the assertions for a method square root are along the lines of the input and output both be non negative numbers.
Unitests on the other hand may test that square root of 9 is 3 and that square root of some negative number yields an exception.
Case in point:
1) One particular Test Method tests a low-level feature. Let's say it tests an instance factory.
2) A LOT of functions depend on this one low-level thing.
Now, the low-level thing is broken, and its Test Method fails.
Do I want to see it buried under hundreds of other failed tests of functionality that relies on it? Or do I want to see that one primary failure?
Just an instance of where I think you'd want to have some ordering of test method execution, namely: lower levels before higher ones.
Note that I do NOT imply a dependency for the order of Test Methods execution. No test method needs a result or state from any other test.
Personally I'd rather have all the error messages to warn me of the scope of the problem - if you run your unit-tests frequently, you'll know what you changed to break them.
I would mock the "instance factory" in the higher level test. If one feature is breaking, preferably only the tests that test that feature should fail. You might break that feature without knowing why and then you want to see the exact feature test fail. Otherwise you might have a hard time figuring out what was broken. You can of course undo the last change to fix it, but you might still not know why everything breaks.
I have a .NET application with a web front-end, WCF Windows service back-end. The application is fairly simple - it takes some user input, sending it to the service. The service does this - takes the input (Excel spreadsheet), extracts the data items, checks SQL DB to make sure the items are not already existing - if they do not exist, we make a real-time request to a third party data vendor and retrieve the results, inserting them into the database. It does some logging along the way.
I have a Job class with a single public ctor and public Run() method. The ctor takes all the params, and the Run() method does all of the above logic. Each logical piece of functionality is split into a separate class - IParser does file parsing, IConnection does the interaction with the data vendor, IDataAccess does the data access, etc. The Job class has private instances of these interfaces, and uses DI to construct the actual implementations by default, but allows the class user to inject any interface.
In the real code, I use the default ctor. In my unit tests for the Run() method, I use all mock objects creating via NMock2.0. This Run() method is essentially the 'top level' function of this application.
Now here's my issue / question: the unit tests for this Run() method are crazy. I have three mock objects I'm sending into the ctor, and each mock object sets expectations on themselves. At the end I verify. I have a few different flows that the Run method can take, each flow having its own test - it could find everything is already in the database and not make a request to vendor... or an exception could be thrown and the job status could be set to 'failed'... OR we can have the case where we didn't have the data and needed to make the vendor request (so all those function calls would need to be made).
Now - before you yell at me and say 'your Run() method is too complicated!' - this Run method is only a mere 50 lines of code! (It does make calls to some private function; but the entire class is only 160 lines). Since all the 'real' logic is being done in the interfaces that are declared on this class. however, the biggest unit test on this function is 80 lines of code, with 13 calls to Expect.BLAH().. _
This makes re-factoring a huge pain. If I want to change this Run() method around, I have to go edit my three unit tests and add/remove/update Expect() calls. When I need to refactor, I end up spending more time creating my mock calls than I did actually writing the new code. And doing real TDD on this function makes it even more difficult if not impossible. It's making me think that it's not even worth unit testing this top level function at all, since really this class isn't doing much logic, it's just passing around data to its composite objects (which are all fully unit tested and don't require mocking).
So - should I even bother testing this high level function? And what am I gaining by doing this? Or am I completely misusing mock/stub objects here? Perhaps I should scrap the unit tests on this class, and instead just make an automated integration test, which uses the real implementations of the objects and Asserts() against SQL Queries to make sure the right end-state data exists? What am I missing here?
EDIT: Here's the code - the first function is the actual Run() method - then my five tests which test all five possible code paths. I changed it some for NDA reasons but the general concept is still there. Anything you see wrong with how I'm testing this function, any suggestions on what to change to make it better? Thanks.
I guess my advice echos most of what is posted here.
It sounds as if your Run method needs to be broken down more. If its design is forcing you into tests that are more complicated than it is, something is wrong. Remember this is TDD we're talking about, so your tests should dictate the design of your routine. If that means testing private functions, so be it. No technological philosophy or methodology should be so rigid that you can't do what feels right.
Additionally, I agree with some of the other posters, that your tests should be broken down into smaller segments. Ask yourself this, if you were going to be writting this app for the first time and your Run function didn't yet exist, what would your tests look like? That response is probably not what you have currently (otherwise you wouldn't be asking the question). :)
The one benefit you do have is that there isn't a lot of code in the class, so refactoring it shouldn't be very painful.
EDIT
Just saw you posted the code and had some thoughts (no particular order).
Way too much code (IMO) inside your SyncLock block. The general rule is to keep the code to a minimal inside a SyncLock. Does it ALL have to be locked?
Start breaking code out into functions that can be tested independently. Example: The ForLoop that removes ID's from the List(String) if they exist in the DB. Some might argue that the m_dao.BeginJob call should be in some sort of GetID function that can be tested.
Can any of the m_dao procedures be turned into functions that can tested on their own? I would assume that the m_dao class has it's own tests somewhere, but by looking at the code it appears that that might not be the case. They should, along with the functionality in the m_Parser class. That will relieve some of the burden of the Run tests.
If this were my code, my goal would be to get the code to a place where all the individual procedure calls inside Run are tested on their own and that the Run tests just test the final out come. Given input A, B, C: expect outcome X. Give input E, F, G: expect Y. The detail of how Run gets to X or Y is already tested in the other procedures' tests.
These were just my intial thoughts. I'm sure there are a bunch of different approaches one could take.
Two thoughts: first you should have an integration test anyway to make sure everything hangs together. Second, it sounds to me like you're missing intermediate objects. In my world, 50 lines is a long method. It's hard to say anything more precise without seeing the code.
The first thing I would try would be refactroing your unit tests to share the set up code between tests by refactoring to a method that sets up the mocks and expectations. Parameterize the method so your expectations are configurable. You may need one or perhaps more of these set up methods depending on how much alike the set up is from test to test.
So - should I even bother testing this
high level function?
Yes. If there are different code-paths, you should be.
And what am I gainging by doing this?
Or am I completely mis-using mock/stub
objects here?
As J.B. pointed out (Nice seeing you at AgileIndia2010!), Fowler's article is recommended read. As a gross simplification: Use Stubs, when you don't care about the values returned by the collaborators. If you the return values from the collaborator.call_method() changes the behavior(or you need non trivial checks on args, computation for return values), you need mocks.
Suggested refactorings:
Try moving the creation and injection of mocks into a common Setup method. Most unit testing frameworks support this; will be called before each test
Your LogMessage calls are beacons - calling out once again for intention revealing methods. e.g. SubmitBARRequest(). This will shorten your production code.
Try n move each Expect.Blah1(..) into intention revealing methods.
This will shorten your test code and make it immensely readable and easier to modify. e.g.
Replace all instances of
.
Expect.Once.On(mockDao) _
.Method("BeginJob") _
.With(New Object() {submittedBy, clientID, runDate, "Sent For Baring"}) _
.Will([Return].Value(0));
with
ExpectBeginJobOnDAO_AndReturnZero(); // you can name it better
on whether to test such function: you said in a comment
" the tests read just like the actual
function, and since im using mocks,
its only asserting the functions are
called and sent params (i can check
this by eyeballing the 50 line
function)"
imho eyeballing the function isn't enough, haven't you heard: "I can't believe I missed that!" ... you have a fair amount of scenarios that could go wrong in that Run method, covering that logic is a good idea.
on tests being brittle: try having some shared methods that you can use in the test class for the common scenarios. If you are concerned about a later change breaking all the tests, put the pieces that concerned you in specific methods that can be changed if needed.
on tests being too long / hard to know what's in there: don't test single scenarios with every single assertion that its related to it. Break it up, test stuff like it should log x messages when y happens (1 test), it should save to the db when y happens (another separate test), it should send a request to a third party when z happens (yet another test), etc.
on doing integration/system tests instead of these unit tests: you can see from your current situation that there are plenty of scenarios & little variations involved in that part of your system. That's with the shield of replacing yet more logic with those mocks & the ease of simulating different conditions. Doing the same with the whole thing will add a whole new level of complexity to your scenario, something that is surely unmanageable if you want to cover a wide set of scenarios.
imho you should minimize the combinations that you are leaving for your system tests, exercising some main scenarios should already tell you that a Lot of the system is working correctly - it should be a lot about everything being hooked correctly.
The above said, I do recommend adding focused integration tests for all the integration code you have that might not be currently covered by your tests / since by definition unit tests don't get there. This exercises specifically the integration code with all the variations you expect from it - the corresponding tests are much simpler than trying to reach those behaviors in the context of the whole system & tell you very quickly if any assumptions in those pieces is causing trouble.
If you think unit-tests are too hard, do this instead: add post-conditions to the Run method. Post-conditions are when you make an assertion about the code. For example, at the end of that method, you may want some variable to hold a particular value or one value out of some possible choices.
After, you can derive your pre-conditions for the method. This is basically the data type of each parameter and the limits and constraints on each of those parameters (and on any other variable initialized at the beginning of the method).
In this way, you can be sure both the input and output are what is desired.
That probably still won't be enough so you will have to look at the code of the method line by line and look for large sections that you want to make assertions about. If you have an If statement, you should check for some conditions before and after it.
You won't need any mock objects if you know how to check if the arguments to the object are valid and you know what range of outputs are desired.
Your tests are too complicated.
You should test aspects of your class rather than writing a unittest for each member of yor class. A unittest should not cover the entire functionality of a member.
I'm going to guess that each test for Run() set expectations on every method they call on the mocks, even if that test doesn't focus on checking every such method invocation. I strongly recommend you Google "mocks aren't stubs" and read Fowler's article.
Also, 50 lines of code is pretty complex. How many codepaths through the method? 20+? You might benefit from a higher level of abstraction. I'd need to see code to judge more certainly.