As we know, TDD means "write the test first, and then write the code". And when it comes to unit-testing, this is fine, because you are limited within the "unit".
However when it comes to UI, writing functional tests beforehand makes less sense (to me). This is because the functional tests have to verify a (possibly long) set of functional requirements. This may usually span multiple screens (pages), preconditions like "logged in", "having recently inserted a record", etc.
According to Wikipedia:
Test-driven development is difficult to use in situations where full functional tests are required to determine success or failure. Examples of these are user interfaces, programs that work with databases, and some that depend on specific network configurations.
(Of course, Wikipedia is no "authority", but this sounds very logical.)
So, any thoughts, or better - experience, with functional tests-first for UI, and then code. Does it work? And is it "pain"?
Try BDD, Behavior Driven Development. It promotes writing specification stories which are then executed step by step stimulating the app to change it's state and verify the outcomes.
I use BDD scenarios to write UI code. Business requests are described using BDD stories and then the functionality is being written to turn the stories i.e. tests green.
The key to testing a UI is to separate your concerns - the behavior of your UI is actually different than the appearance of your UI. We struggle with this mentally, so as an exercise take a game like Tetris and imagine porting it from one platform (say PC) to another (the web). The intuition is that everything is different - you have to rewrite everything! But in reality all this is the same:
The rules of the game.
The speed of the blocks falling.
The logic for rows matching
Which block is chosen
And more...
You get the idea. The only thing that changes is how the screen is drawn. So separate how your UI looks from how it works. This is tricky, and usually can't be perfect, but it's close. My recommendation is to write the UI last. Tests will provide the feedback if your behavior is working, and the screen will tell if it looks right. This combination provides the fast feedback we're looking for from TDD without a UI.
TDD for functional tests makes sense to me. Write a functional test, see it failing, then divide the problem into parts, write a unit test for each part, write code for each part, see unit tests pass and then your functional test should pass.
Here is a workflow suggested in the book Test-Driven Development with Python by Harry Percival (available online for free):
P.S. You can automate your functional tests using e.g. Selenium. You can add them to the Continuous Integration cycle as well as unit tests.
ATDD is very useful if you have a very good grasp on how the UI should behave and also ensure that the cost of change to the functional tests that you build upfront is less.
The biggest problem with doing this, of course, is most often the UI is not fully speced out.
For example, if you are building a product, and are still doing rapid iterations of getting a UI that goes through a usability testing and the feedback incorporated, you do not want the baggage of fixing your functional tests with every small change to the UI.
Given that functional tests are generally slow, the feedback cycle is high and its very painful to keep them green with the changes to UI.
I work on a product team where we had this exact same decision to make. A colleague of mine has summarized our final approach very well here. (Disclaimer: Please ignore the tool specific details there.)
I've done Acceptance TDD with a UI. We would assert the common header and footer were used via xpath'ing for the appropriate ids. We also used xpath to assert the data appeared in the correct tag relative to whatever ids we used for basic layout and structure. We would also assert the output page is valid html 4.01 strict.
Programmatic UI tests is a salvation when you want to be confident your UI does what expected. The UI tests are closer to BDD (behavior driven development) than to TDD. But the terminology is cloudy and however you call 'em they are useful!
I have got rather good experience using cucumber. I use it to test flex applications but it's mostly used to test web apps. Click the link! The site have got really nice examples of methodology.
Advanced UI is not compatible with straight TDD.
Use XP practices to guide you.
It is unwise to slavishly follow TDD (or even BDD) for UI, it's worth thinking about the root goals of both practices. In particular, we want to avoid using general purpose TDD tools for UI. (If) They're simply not designed for that use case, you end up binding yourself to tests that you have to iterate on as rapidly as the UI code (I call this, UI test-lock).
That's not the purpose of TDD. We don't do it to go slower, we use to go fast (and NOT break things!).
We want to achieve a few things with TDD:
Code / Algorithmic Correctness
Minimal code to complete a test
Fast feedback
Architectural separation of concerns to the units (modular parts under test, no more, no less)
Formalize specification in a useful/usable way
To some extent, document what the module/system does (e.g. provide an example of use.)
We can achieve these goals in UI development, but how and where we apply practices is important to avoid test-lock.
Applying these principles to UI
What benefits can we get from testing UI? How can we avoid test-lock.
With usual TDD one major benefit is that we can think about the abstractions, and generally design the shape of a test subject, as we think of names, relationships etc.
For UI, much of what we want to do, especially if it's non-trivial, is only reasoned about effectively when we can see it manifested on screen.
Writing tests that describe what we expect to see in a UI, specifically what data, can be TDD tested. For example, if we have a view that should show an account balance, a test that expects it to appear in an on-screen element that can be addressed by some form of ID, is good. We will know our app/view is displaying content that's expected, in standard UI library elements.
If, on the other hand, we want to create a more complex UI and wish to apply TDD to it, we will have some problems. Chances are, we won't even know how to code it, if it's novel enough.
For this we would prototype and iterate with tools which give us fast feedback, but don't saddle us with any need to write tests first. When we reach a point where implementation becomes clear, we can then switch up to test first for the production code.
This keeps us fast. Designers and Product owners in the team can also see results, provide inputs, and tweaks.
The code should be well-formed. As soon as we identify small potential pieces which will make it to production code, we can migrate them to be under test. Using TCR or a similar method to add tests, or simply use the TDD method to re-write. Remember once it works, you can apply snapshot/record-playback testing, to keep regression errors at bay.
Do what works, keep it simple.
Addendum
Notes about the scope of UI discussed above
There will be a need to figure out how to make things work as expected in any advanced UI. There is also a likelihood that specs change and tweak in extremely arbitrary / capricious ways. We need to ensure any tests we apply to these test subjects are flexible, or extremely easy to re-generate based on a working model. (replay tests are gold for this.)
Good development practices, in a nutshell, code separation. Is going to help you ensure the code is at least testable, and simpler to debug, maintain and reason about.
Don't be a slave to ritual.
It doesn't make you correct. Just slower.
Related
What strategy would you recommend to unit test an existing Grails app?
I've just read Beck Kent's book on TDD and would like to apply similar approach to my app.
My aim is to unit test the entire code base and to be able to refactor the code and make it "cleaner". By "cleaner" i mean I want to reduce duplication, make my controllers slimmer by extracting common logic into services and so on.
So where should I start? Models? Controllers?
What is your "bad" and "good" experience doing similar thing?
#Péter.
My app is not too big in my opinion. it consists of 12+ models, similar amount of controllers, few services and around 15 utils classes.
One of the main reasons I want to have full unit test coverage is that in many cases the system just works. While it's good from the user point of view from developer point of view such code is a nightmare to change and maintain.
Another important thing I would like to make small and fast regular releases (new little features and/or improvements), but without unit test coverage it would be almost imposible.
So the question is not: "Do i need to do it?", but "How can I do it?"
Depends on the size of the app, but for any nontrivial real life app it is a huge effort to satisfiably cover it with unit tests. So you need to prioritize your efforts, to focus on the most critical / most frequently changed / most buggy parts of the system (usually these overlap quite a bit: the most critical parts are usually the ones most often touched to add new functionality or to fix bugs).
A good method is to write unit tests more or less in TDD fashion whenever you need to touch any part of the code. I wrote "more or less" because for legacy code, you typically need to write much higher level, more complex unit tests than for greenfield development. In fact, in some cases it may not even be productive to start with unit tests, instead it is better to create functional/system tests to cover large grained functionality from the user perspective.
Note that, depending on the available documentation and level of developer/user knowledge about the system, you may not always be able to determine the "correct" behaviour for a specific functionality, only its current behaviour. Even in this case, it is worth covering it with (unit) tests: these document the code's actual behaviour and detect any unexpected changes in it in the future.
Once you have the actual piece of code reasonably covered with unit tests, this gives you the confidence needed for refactoring. Do some (simple or more complex) refactoring whenever you touch the code. However, don't overdo it. If you need to change a single line for a bugfix, it may be overkill to start refactoring a whole inheritance hierarchy (even if it is really messy). Make notes about such impending refactoring tasks and try to schedule them later.
I generally agree with #Peter in that this is a non-trivial thing to do, and you might be hampered by just knowing what things do, not what they are supposed to do. But thats ok, if not ideal, because testing is just as much about maintainability (i.e. knowing when changes break things) as it is about correctness. So put tests around the existing functionality as it is right now, and if you have to fix a bug, you can change your test to capture the fix.
If you and your team is focused on adding new features, then implement TDD now, i.e. test the new functionality, and add in tests for the existing functionality you are modifying, as was already suggested.
If you can focus on testing, then I would test in vertical layers. Write some tests for a model, then its services, then the controllers (that order is just a suggestion). The point is if you are testing the 'Book' part of your app, test all Book functionality before moving on. Also, you can prioritize on the core functionality of the app. If 'Books' are much more important than something else, test 'Books' first.
Personally I found it easiest to do Grails integration testing. Only for some commonly used beans, I did unit testing. It's not very TDD I admit, but still gives you some confidence.
I am very new to test-driven development (TDD), not yet started using it.
But I know that we have to write tests first and then the actual code to pass the test and refactor it till the design is good.
My concern over TDD is where it fits in our systems development life cycle (SDLC).
Suppose I get a requirement of making an order processing system.
Now, without having any model or design for this system, how can I start writing tests?
Shouldn't we require to define the entities and their attributes to proceed?
If not, is it possible to develop a big system without any design?
There is two levels of TDD, ATDD or acceptance test driven development, and normal TDD which is driven by unit tests.
I guess the relationship between TDD and design is influenced by the somewhat "agile" concept that source code IS the design of a software product. A lot of people reinforce this by translating TDD as Test Driven Design rather than development. This makes a lot of sense as TDD should be seen as having a lot more to do with driving the design than testing. Having acceptance and unit tests at the end of it is a nice side effect.
I cannot really say too much about where it fits into your SDLC without knowing more about it, but one nice workflow is:
For every user story:
Write acceptance tests using a tool like FitNesse or Cucumber, this would specify what the desired outputs are for the given inputs, from a perspective that the user understands. This level automates the specifications, or can even replace specification documentation in ideal situations.
Now you will probably have a vague idea of the sort of software design you might need as far as classes / behaviour etc goes.
For each behaviour:
Write a failing test that shows how calling code you would like to use the class.
Implement the behaviour that makes the test pass
Refactor both the test and actual code to reflect good design.
Go onto the next behaviour.
Go onto the next user story.
Of course the whole time you will be thinking of the evolving high level design of the system. Ideally TDD will lead to a flexible design at the lower levels that permits the appropriate high design to evolve as you go rather than trying to guess it at the beginning.
It should be called Test Driven Design, because that is what it is.
There is no practical reason to separate the design into a specific phase of the project. Design happens all the time. From the initial discussion with the stakeholder, through user story creation, estimation, and then of course during your TDD sessions.
If you want to formalize the design using UML or whatever, that is fine, just keep in mind that the code is the design. Everything else is just an approximation.
And remember that You Aren't Gonna Need It (YAGNI) applies to everything, including design documents.
Writing test first forces you to think first about the problem domain, and acts as a kind of specification. Then in a 2nd step you move to solution domain and implement the functionality.
TDD works well iteratively:
Define your initial problem domain (can be small, evolutionary prototype)
Implement it
Grow the problem domain (add features, grow the prototype)
Refactor and implement it
Repeat step 3.
Of course you need to have a vague architectural vision upfront (technologies, layers, non-functional requirement, etc.). But the features that bring added-value to your your application can be introduced nicely with TDD.
See related question TDD: good for a starter?
With TDD, you don't care much about design. The idea is that you must first learn what you need before you can start with a useful design. The tests make sure that you can easily and reliably change your application when the time comes that you need to decide on your design.
Without TDD, this happens: You make a design (which is probably too complex in some areas plus you forgot to take some important facts into account since you didn't knew about them). Then you start implementing the design. With time, you realize all the shortcomings of your design, so you change it. But changing the design doesn't change your program. Now, you try to change your code to fit the new design. Since the code wasn't written to be changed easily, this will eventually fail, leaving you with two designs (one broken and the other in an unknown state) and code which doesn't fit either.
To start with TDD, turn your requirements into test. To do this, ask "How would I know that this requirement is fulfilled?" When you can answer this question, write a test that implements the answer to this question. This gives you the API which your (to be written) code must adhere to. It's a very simple design but one that a) always works and b) which is flexible (because you can't test unflexible code).
Also starting with the test will turn you into your own customer. Since you try hard to make the test as simple as possible, you will create a simple API that makes the test work.
And over time, you'll learn enough about your problem domain to be able to make a real design. Since you have plenty of tests, you can then change your code to fit the design. Without terminally breaking anything on the way.
That's the theory :-) In practice, you will encounter a couple of problems but it works pretty well. Or rather, it works better than anything else I've encountered so far.
Well of course you need a solid functional analysis first, including a domain model, without knowing what you'll have to create in the first place it's impossible to write your unit tests.
I use a test-driven development to program and I can say from experience it helps create more robust, focussed and simpler code. My recipe for TDD goes something likes this:
Using a unit-test framework (I've written my own) write code as you wish to use it and tests to ensure return values etc. are correct. This ensures you only write the code you're actually going to use. I also add a few more tests to check for edge cases.
Compile - you will get compiler errors!!!
For each error add declarations until you get no compiler errors. This ensures you have the minimum declarations for your code.
Link - you will get linker errors!!!
Write enough implementation code to remove the linker errors.
Run - you unit tests will fail. Write enough code to make the test succeed.
You've finished at this point. You have written the minimum code you need to implement your feature, and you know it is robust because of your tests. You will also be able to detect if you break things in the future. If you find any bugs, add a unit test to test for that bug (you may not have thought of an edge case for example). And you know that if you add more features to your code you won't make it incompatible to existing code that uses your feature.
I love this method. Makes me feel warm and fuzzy inside.
TDD implies that there is some existing design (external interface) to start with. You have to have some kind of design in mind in order to start writing a test. Some people will say that TDD itself requires less detailed design, since the act of writing tests provides feedback to the design process, but these concepts are generally orthogonal.
You need some form of specification, rather than a form of design -- design is about how you go about implementing something, specification is about what you're going to implement.
Most common form of specs I've seen used with TDD (and other agile processes) are user stories -- an informal kind of "use case" which tends to be expressed in somewhat stereotyped English sentences like "As a , I can " (the form of user stories is more or less rigid depending on the exact style/process in use).
For example, "As a customer, I can start a new order", "As a customer, I can add an entry to an existing order of mine", and so forth, might be typical if that's what your "order entry" system is about (the user stories would be pretty different if the system wasn't "self-service" for users but rather intended to be used by sales reps entering orders on behalf of users, of course -- without knowing what kind of order-entry system is meant, it's impossible to proceed sensibly, which is why I say you do need some kind of specification about what the system's going to do, though typically not yet a complete idea about how it's going to do it).
Let me share my view:
If you want to build an application, along the way you need to test it e.g check the values of variables you create by code inspection, of quickly drop a button that you can click on and will execute a part of code and pop up a dialog to show the result of the operation etc. on the other hand TDD changes your mindset.
Commonly, you just rely on the development environment like visual studio to detect errors as you code and compile and somewhere in your head, you know the requirement and just coding and testing via button and pop ups or code inspection. this is a Syntax debugging driven development . but when you are doing TDD, is a "semantic debugging driven development " because you write down your thoughts/ goals of your application first by using tests (which and a more dynamic and repeatable version of a white board) which tests the logic (or "semantic") of your application and fails whenever you have a semantic error even if you application passes syntax error (upon compilation).
In practice you may not know or have all the information required to build the application , since TDD kind of forces you to write tests first, you are compelled to ask more questions about the functioning of the application at a very early stage of development rather than building a lot only to find out that a lot of what you have written is not required (or at lets not at the moment). you can really avoid wasting your precious time with TDD (even though it may not feel like that initially)
So - management is looking to do a push to move towards doing unit-testing in all the applications moving forward - and eventually get into full TDD/Continuous Integration/Automated build mode (I hope). At this point however we are just concerned about getting everyone developing apps moving forward using unit-testing. I'd like to just start with the basics.
I won't lie - I'm by far no expert by any means in unit-testing, but I do have a good enough understanding to start the initiative with the basics, and allow us to grow togeather as a team. I'd really love to get some comments & critisism from all you experts on my plan of attack on this thing. It's a team of about 10 developers in a small shop, which makes for a great opportunity to move forward with agile development methodologies and best practices.
First off - the team consists of mainly mid level developers with a couple of junior devs and one senior, all with minimal to no exposure to unit testing. The training will be a semi-monthly meeting for about 30-60 minutes each time (probably wind up running an hour long i'd guess, and maybe have them more often). We will continue these meetings until it makes sense to stop them to allow others to catch up with their own 'homework' and experience - but the push will always be on.
Anyway - here is my lesson plan I have come up with. Well, the first two at least. Any advice from your experts out there on the actual content or structure of the lessons, etc, would be great. Comments & Critisism greatly appreciated. Thanks very much.
I apologize if this is 'too much' to post in here or read through. I think this would be a great thread for SO users looking to get into unit testing in the first place as well. Perhaps you could just skip to the 'lesson plans' section - thanks again everyone.
CLIFF NOTES - I realize this post is incredibly long and ugly, so here is the cliff notes - Lesson 1 will be 'hello world unit tests' - Lesson 2 will be opening the solution to my most recent application, and showing how to apply each 'hello world' example in real life... Thanks so much everyone for the feedback you've given me so far.. just wantd to highlight the fact that lesson 2 is going to have real life production unit tests in it, since many suggested I do that when it was my plan from the begining =)
Unit Testing Lesson Plan
Overview
Why unit test? Seems like a bunch of extra work - so why do it?
• Become the master of your own destiny. Most of our users do not do true UATs, and unfortunately tend to do their testing once in production. With unit-tests, we greatly decrease risk associated with this, especially when we create enough test data and take into account as many top level inputs as we possibly can. While not being a ‘silver bullet’ that prevents all bugs – it is your first line of defense – a huge front line, comparable to that of the SB championship Giants.
• Unit-Testing enforces good design and architecture practices. It is ‘the violent psychopath who maintains your code and knows where you live’ so to say. You simply can’t write poor quality code that is well unit-tested
• How many times have you not refactored smelly code because you were too scared of breaking something? Automated testing remove this fear, makes refactoring much easier, in turn making code more readable and easier to maintain.
• Bottom line – maintenance becomes much easier and cheaper. The time spent writing unit tests might be costly now – but the time it saves you down the road has been proven time and time again to be far more valuable. This is the #1 reason to automate testing your code. It gives us confidence that allows us to take on more ambitious changes to systems that we might have otherwise had to reduce requirements on, or maybe even not take on at all.
Terminology Review
• Unit testing - testing the lowest level, single unit of work. E.G. – test all possible code paths that a single function can flow through.
• Integration testing - testing how your units work together. E.g. – run a ‘job’ (or series of function calls) that does a bunch of work, with known inputs - and then query the database at the end and assert the values are what you expect from those known inputs (instead of having to eye-ball a grid on a web page somewhere, e.g. doing a functional test).
• Fakes – a fake is a type of object whose purpose is to use for your testing. It allows you too easily not test code that you do not want to test. Instead of having to call code that you do not want – like a database call – you use a fake object to ‘fake’ that DB call and perhaps read the data from an XML/Excel file or a mocking framework.
o Mock – a type of fake to which you make assert statements against.
o Stub – a type of fake to which you use as placeholder code, so you can skip the database call, but do not make asserts against
Lessons
Lesson one – Hello Worlds
• Hello World unit test - I will create a ‘hello world’ console application that is unit tested. Will create this application on the fly during the meeting, showing the tools in Visual Studio 2008 (test-project, test tools toolbar, etc.) that we are going to use along the way while explaining what they do. This will have only a single unit-test. (OK, maybe I won’t create it ‘on the fly’ =), have to think about it more). Will also explain the Assert class and its purpose and the general flow of a unit-test.
• Hello World, a bit more complicated. Now our function has different paths/logical branches the code can flow through. We will have ~3 unit tests for this function. This will be a pre-written solution I make before the meeting.
• Hello World, dependency injection. (Not using any DI frameworks). A pre-written solution that builds off the previous one, this time using dependency injection. Will explain what DI is and show a sample of how it works.
• Hello World, Mock Objects. A pre-written solution that builds off the previous one, this time using our newly added DI code to inject a mock object into our class to show how mocking works. Will use NMock2.0 as this is the only one I have exposure to. Very simple example to just display the use of mock objects. (Perhaps put this one in a separate lesson?).
• Hello World, (non-automated) Integration Test. Building off the previous solution, we create an integration test so show how to test 2 functions together, or entire class together (perhaps put this one in a separate lesson?)
Lesson two – now we know the basics – how to apply this in real life?
• General overview of best practices
o Rule #1- Single Responsibility Principal. Single Responsibility Principal. Single Responsibility Principal. Facetiously stating that this is the single most important thing to keep in mind while writing your code. A class should have only one purpose; a function should do only one thing. The key word here is ‘unit’ test – and the SRP will keep your classes & functions encapsulated into units.
o Dependency Injection is your second best friend. DI allows you to ‘plug’ behavior into classes you have, at run time. Among other things, this is how we use mocking frameworks to make our bigger classes more easily testable.
o Always think ‘how will I test this’ as you are writing code. If it seems ‘too hard to test’, it is likely that your code is too complicated. Re-factor until it into more logical units of classes/functions - take that one class that does 5 things and turn it into 5 classes, one which calls the other 4. Now your code will be much easier to test – and also easier to read and refactor as well.
o Test behavior, not implementation. In nutshell, this means that we can for the most part test only the Public functions on our classes. We don’t care about testing the private ones (implementation), because the public ones (behavior) are what our calling code uses. For example... I’m a millionaire software developer and go to the Aston Martin dealership to buy myself a brand new DB9. The sales guy tells me that it can do 0-60 in 3 seconds. How would you test this? Would you lift the engine out and perform diagnostics tests, etc..? No... You would take it onto the parkway and do 160 MPH =). This is testing behavior vs. implementation.
• Reviewing a real life unit-tested application. Here we will go over each of the above ‘hello world’ examples – but the real life versions, using my most recent project as an example. I'll open a simple unit test, a more complex one, one that uses DI, and one that uses Mocks (probably coupled to the DI one). This project is fairly small & simple so it is really a perfect fit. This will also include testing the DAL and how to setup a test database to run these tests against.
I like the commitment and thoroughness you're expressing here, but my feeling is your adoption plan is going to take longer this way than if you started straight in, paired up and wrote some actual production tests. You'll need infrastructure anyway -- why not bootstrap your actual production code with a CI server and a TDD framework? (A lot of times this is a bigger pain point than learning "asserts." Get this part out of the way sooner rather than later). Then sit down and solve an actual problem with a fellow coder. Pick a simple bug or small feature and try to identify what the failing tests would look like and try to write them.
I like Hello Worlds for a brown bag lunch or something. But I can't think of any reason not to just jump right in and solve some problems.
I helped design and run a Test Driven Development training in house at my company for Java developers. We ended up doing it as a full day training, and we broke it up similar to how you have here.
Why do testing?
Simple example.
More complex example.
One thing I must stress though, is people need to learn by doing.
For my training, we set up a lab environment where each student could start with the same snapshot of code, and develop and run the tests themselves, with instructors walking around the training room helping individuals who were confused or weren't getting it.
For the "Simple Example" we had a "cooking show" version of code that was on a projector and went through the TDD process step by step. Developers would have to go through the process of writing a test, then creating implementation that was just sufficient to pass the test, then repeat. At each phase, a prepared solution to the current phase was shown on the projector after the students had some time to try it on their own.
For the "Complex Example" we created a set of requirements, and then allowed the students to come up with their own solutions using TDD to do so. We even had an exercise where requirements surprise, surprise changed suddenly part way through the exercise.
I like your idea of doing it over a longer period of time with regular checkups. One downfall of our training was a lack of followup. Developers benefited by the training, I'm sure, but many I think, did not take the practice back to their ordinary work. Regular checkups would help instill unit testing as a matter of habit.
For more ideas, check out my answer to this question.
Getting people interested in unit testing on someone else's/sample code will not be as effective as having folks "start with the testcase" on some new code they have to write.
Cover the basics of starting with a test case and triangulating to a valid result (all the while emphasizing that a failing test is progress).
That said, my personal experience is that things like this are better done in a pair programming environment. It's extra work for you, but the combination of one-on-one, some ownership and a working example they can reference at the end of it will make adoption much easier.
Getting a coverage tool running at some point later on can, given the right environment, provide a fun, competitive and gratifying way of marking progress. Making this in any way part of compensation/bonuses is a real bad idea. Done correctly, folks will adopt it because it works.
I tried to introduce TDD in a small team (7-9 programmers). Lectures failed. People are skeptical, TDD sounds like snake oil. Plus, there's always the proverbial Willy.
In the end, instead of TDD, people on the team do occasional DDT (Development-Driven Tests): the write unit tests after the code to confirm that it does what they meant it to. Looks like utter failure until you realize that even this form of developer-driven QA is much better than what you had before because, even if it doesn't really improve the code for the future revisits, it cuts into deployed bugs.
There was, however, one key difference compared to your setting: the push didn't come from the management, they were just good enough to let the programmers decide for themselves.
The lectures failed, but I didn't put all the eggs in a single basket. I wrote a bunch of tests for many of the smaller, more focused functions and classes used across the project. Those little buggers you occasionally need to bend a bit to fit exactly in the next call site, hoping you didn't break any of the existing ones (you looked around the code, the change seemed safe). It was running the tests after every other svn up (not only by me), and the subsequent replies to the guilty commit emails "you broke it, pls fix ASAP!" that finally convinced them of at least partial utility of unit tests.
No matter what examples you come up with for the lectures, they'll tell you it only worked because the CUT was unrealistically (or unusually) isolated from the rest of the code. They'll ask you to write unit tests for a convoluted, buggy God Class (preferably a Singleton, everybody loves global variables, no?) because, to the best of their knowledge, that's what real-world code looks like, and they think it'll look like that with TDD as well.
Screw lectures. Have the management buy everybody a few books for the Christmas: at the very least TDD By Example (Beck) and Refactoring (Fowler). Speaking from experience, the books won't necessarily have any impact with most of them (more of the "that wasn't real-world enough" bullshit), but it's bound to stick to someone, and, no offense intended, I'd bet $1000 that Beck is the better evangelist of you two. ;) Practice TDD yourself in code you share with them. If it really works, they'll follow. Sss-looowww-lyyyy, but they will.
Then again, maybe you don't have that many Willies on your team, maybe your colleagues are eager, just in need of a leader, I don't really know, you didn't tell.
First commandment: don't be put off by slow and/or partial intake: every inch counts! (I'm preaching to myself here, ya know...)
I think it will be harder to start with test-after type of testing. A major difference compared to test-first is that test-after does not guide you in writing code which is easy to test. Unless you have first done TDD, it will be very hard to write code for which it's easy to write tests.
I recommend starting with TDD/BDD and after getting the hang of that, continue learning how to do test-after on legacy code (also PDF). IMO, doing test-after well is much harder than test-first.
In order to learn TDD, I recommend finishing this tutorial. That should get you started on what kind of tests to write.
I think it's a good outline for a session or two. But I'd strongly recommend standing up a testing infrastructure first and setting the developers up for early success, rather than shoving the developers head-first into the unit testing world and shouting "ready, set, GO!"
Assuming you develop in an IDE, a one-click test generator tool will really help with step one, test creation. You'll want to integrate a GUI-based test runner with your IDE, and you really need an automated test runner integrated with your CI builds. Having a code coverage tool integrated with the test runner tool will help developers increase their coverage. You need to provide developers with a complete framework for writing the tests, and that includes documenting how to organize the test source code (test naming standards, project settings, folder locations, etc.) Tools that automate any of the manual steps will go a long way. Provide a mock framework including mock objects for any in-house or third-party library APIs you depend on. If you're going to recommend a database full of fake data (instead of a suite of fake data access objects) create one with a few key tables and populate it with test data, and provide whatever it takes to reset it between testing runs. If not, provide a few fake DAOs of your own to serve as examples. You should have a small suite of tests already in your source control system testing production code to show as examples. And you need to have all this documented so you can hand it out in meeting #1. (Don't forget to test the document!)
One of our teams tried the ad hoc approach a couple years ago, but adoption was poor. Without the definitions or the organization of the tests, individual developers kind of did their own thing. Lack of a naming standard made it difficult to identify the right tests to run for which modules, and few people bothered. We had no IDE integration for any of it. Nobody wanted to write mocks, and we didn't provide mocks in advance for the framework we use. And it's really hard to add tests to legacy code that wasn't written with Dependency Injection in mind.
I'm now trying to get our infrastructure reorganized and ready for another shot at the task. Only after it's standing up will I work on trying to get the developers to use it again.
However, you have the one thing we were kind of lacking last time, and that's strong management support. Our previous managers were kind of lukewarm to the idea because they didn't want to delay coding by having to write all these tests. We now have new management and they are focusing on code quality -- automated unit tests are now in vogue, so it's a good time for us to try again.
Is a classroom style approach going to work well? That's a question that comes to my mind as what may happen here is so much of a separation between knowledge and application of the ideas that while they may know it, they don't quite get how to use it.
Just to give a different approach, one could jump into whatever projects are going on and start trying to apply some of this which may take a while to trickle down through everyone as the idea would be that you'd train one and then as a pair you'd train another pair and so on, to try to get everyone up to the same point. This way the theoretical stuff that isn't relevant gets taken out early.
Another idea would be to see if it makes sense to try to form teams within the group of 10, such as 2 groups of 3 and a group of 4 or 5 pairs, so that each person can present what they've done and how well do they use this.
I'd second the boundary testing, as well as showing why it can be useful to test with invalid input to see that this doesn't cause the system to roll over and die.
Lastly, it may make sense to have time devoted to handling questions and other 1:1 issues that people have as not everyone will want to ask questions in front of a large group.
EDIT: A caution on Lesson 2, which is a good idea but has a few hazards to note. First, be humble in doing this lesson so that you aren't coming across as a show-off wanting to slam everyone else. Second, I'd suggest leaving some tests incomplete when going over things so that there is an interactive aspect here that prevents the other developers from merely tuning out or rolling their eyes. Third, a follow-up lesson or two of others showing what they did or you going over what they did may be more useful to show that this is a team effort in some ways.
If management is serious about this, then they have to take the bull by the horns and take action themselves not just have a little training. Start having them ask to see the tests in code reviews. Once this is common and people know they willnot pass the code reveiw without a test, then have them refuse to deploy new code until there are tests written. It will only take a couple of iterations of this before people will know that they can't get away wihtout writing the tests and then they will do so fairly reliably. Yes the first time the manager refuses to deploy, something will probably be late. They have to take that into accound in their own planning. They also will need to start adding test writing time to their task estimating. No one wants to do this if they think they will be blamed because it takes longer to write tests and code than just to write code. Have a published schedule for how this will be implemented (you need to give them some time to learn how to do this before it is required for all new development) and on the drop dead date, nothing gets to prod without tests.
I like it. One comment I have is it's probably worth dropping in some guidelines/advice on how to go about testing threaded/asynchronous code.
Also, I'd make a comment about boundary testing i.e. using unit tests to state your assumptions on any 3rd party API you utilize.
Just because I have been doing something where TDD (and unit tests in general) has been invaluable and because it might make a good demo, I'd suggest an example using Regex, especially if you aren't a Regex expert, it's quite easy to make mistakes.
It'd also make for a good 'show, not tell' demo. If you just:
step 1: Tell them just to write some code with a Regex to match a certain pattern.
step 2: expand the pattern to something else.
step 3: show how unit tests immediately let them be sure that a) it matches the new pattern and b) it hasn't broken any of the previous matches or let through results that shouldn't be.
In general C&C of your review, it's hard to get an idea of exactly what sequence events will go in. Also, semi-month seems far too infrequently if there's just a void inbetween with no push to apply their knowledge. The last thing you want is it to become some meeting they go to and forget about as soon as they are out of the room!
Do you have sufficient downtime in your current workload to address technical debt and get the team to actually pair up and start looking at adding unit tests to some existing functionality and refactoring where necessary to allow mocking?
I would say that will prove far more effective than lectures and potted workshops, YMMV.
First a little background. The company I work for writes web based software that is a hosted solution for our customers (ie ASP (Application Service Provider)). We are adopting agile practices such as Scrum and we execute sprints to build new features for our product.
I am a proponent of TDD (Test Driven Design), and as a part of what I deliver in a sprint I always write tests and I always get them integrated with the build (ie ccnet); however the other developers do not follow this practice and it is not enforced.
Is it a good practice to force a development group into providing unit tests as a part of what is delivered in a sprint?
Unless you are a position of authority, the best thing you can do is to convince them of the value of the test suite.
It's very difficult to get developers to see the light on this issue if they aren't seeing it done right.
Try to pair with another developer and show them the benefits and the clarity that comes from writing the tests FIRST. If you don't do this, they are likely to write all of their code, get it working, and then write tests. So, from their point of view, it will feel like simply an extra task that doesn't help them get things done.
Also keep in mind that people often do not understand how to write good tests. Even more, some do not know how to make use of tools like jmock, which can lead to them getting stuck and giving up on writing a test.
Forcing anything onto anybody is not a good practise in my view. I would show them the benefits of TDD at every opportunity I get. This should automatically get the rest of the team to voluntarily practise TDD.
You don't want lip-service unit testing, you want whole-hearted unit testing. That isn't something that can be forced. What you need to do is influence your teammates over time to see the benefits of unit testing and to develop a unit testing culture.
To start with you need to understand that different people change for different reasons. In Crossing the Chasm terms, visionaries will adopt new techniques because they are better, but pragmatists adopt new techniques either because they solve a problem/pain the currently have or because everyone else is adopting it.
Your mission then it to show how unit testing can solve a pain your team currently feels. As you win over people one-by-one eventually you can reach a tipping point where unit testing is the norm and everyone goes along with it. However if you can't tie unit testing to a pain your team feels then your efforts to convince them will likely fail.
Considering unit-testing improves code and software quality on a long term, I would say that, yes, it is good practice to have your developpers provide unit-tests -- be it part of some kind of sprint or not.
The two main barriers to unit-tests I've seen are :
the developpers don't get the point : "why would we write more code just to test ?"
"we don't have time to write unit-tests"
To answer the first point, you'll have to provide some sort of demonstration / formation, I suppose ; if you can get developpers to see why unit-tests are useful, they will like them, and use/develop them ; but they need to seen why those are useful : unless you are their boss, you cannot force people to develop unit-tests.
And, even if you are their boss, they will probably not do the best possible job if they are being forced : unit-testing is often done better if people understand why and how !
To answer the second point... Well, you obviously need to get your developpers some "special" time to developp unit-tests ; it can mean less time to do manual testing, btw.
Another thing is : it is hard to know "what to test", and "how to test" : you will need to explain / demonstrate that to your colleagues : some things cannot be tested, some things don't need to be, and some things are not "unit-testable" -- well, I suppose, unless you software is really well engineered ^^
I've gotten in that position on many jobs and contracts in the past, so I've finally gotten discouraged and embraced the darkness by advocating Development Driven Development.
I've found that when most managers embrace XP, they're embracing throwing out the documentation, not really doing TDD. Programmers on most teams are rewarded for quick hacks that get the defect out of their queue, and it's one manager in about ten who has the guts to stand up to senior management as an advocate of the overall quality of the product as opposed to the bottom-line-for-this-quarter way of doing business. After all, most software jobs that pay anything are corporate sponsored, and Freud proved that corporations are insane.
Or at least, he should have.
There is a great Joel on Software article on this called Getting Things Done When You're Only a Grunt. His strategies applied to united testing would be the following:
Just Do It
Most important, I think. Regularly write tests, do not make it appear as if you yourself see them as a minor part of development.
Harness the Power of Viral Marketing
If you see an error in someone else's code, write a test that triggers it and present both to him. Maybe he sees your point.
Create a Pocket of Excellence
Identify the team members who are open to the idea but not quite sure how to work with unit tests and set up a number of test classes until everyone of them knows how to do it. Then turn towards the other ones. Things are a lot easier to establish once you are not the only one anymore.
Neutralize The Bozos
There will be team members who are almost impossible to get to write tests. Maybe those should be dealt with by regularly breaking their code with a new commit - and then pointing out that since they don't have tests it was hard for you to notice.
it depends on your Version Control, but there are Version Control Software that enable you to run scripts before check-in or before merge to the Production branch. and it will not let the developer check in if the unit testing is failed.
First, talk to your manager. If he is convinced that testing is a good thing, add a coverage test to your build system. If the coverage of your unit tests falls below a certain level, you can handle it as a failed build. This gives your colleagues a measure, a way to see when the fail to deliver tested code.
In your case, NCover seems to integrate nicely into CC.NET.
Create a matrix that you want developers to achieve.
Make sure there is a subtask for Unit test creation for every story.
No. In my experience, TDD isn't all that useful in practice. I sometimes use it for really general fundamental classes (like geometry or generic data structures) that lend themselves naturally to automated tests. But for UI components or business logic, I find it's more trouble than it's worth.
My question assumes that folks already believe that unit tests of some sort are worthwhile and actually write them on their current projects. Let's also assume that unit tests for some parts of the code are not worth writing because they're testing trivial features. Examples are getters/setters, and/or things that a compiler/interpreter will catch immediately. The contrary assumption is that "interesting" code is worth testing.
The level of coverage for a region of code should be directly proportional to a combination of the likelihood of change and the amount of functionality that depends on it.
For example, we have some fairly complex parsing logic at the core of our base control classes. We unit-test the absolute piss out of it because it has to handle input which comes from systems outside our influence (high possibility of change) + ALL of our UI controls depend on it to be stable. The slightest tweak in the base ripples and magnifies through layers of inheritance and dependency.
The unit tests that verify data passed between layers often add the most value
UI <----> Business <---> Data
I find that the tests with the most pay-off are the ones testing for previously found bugs. In my opinion, if a team were to do nothing but add regression tests for bugs as they fixed them, they would get more bang for their buck than almost any other testing strategy.
By testing what has been broken (and testing it thoroughly), you know that you're testing areas susceptible to breakage.
This of course assumes that you are testing things that break during development not just waiting for QA reports to trickle in after a deploy or something like that.
As the question is asked, I have no doubt whatsoever that functional tests like the selenium test provides the greatest business value. A selenium (web) test clicks through the application and asserts the expected behavior, as defined by the user story/use cases. These tests assert that the main "thing" your software is trying to communicate to the customer is good.
If these things do not work, your customer will loose confidence in your ability to produce credible software, hence taking away a fair amount of business value.
Now there may be a fair amount of details that these tests do not cover, but I really believe they are essential in the creation of trust, especially when talking to a mass market.
To take a shot at my own question, here's the modules I've found worthwhile testing:
a) Any of the business rules. On my current application the business rules are externalized from the business objects.
b) Any of the virtual attributes on the business objects. Often these virtual attributes have involved algorithms.
c) Major components where we can state the expected produced output for a given input consumed. Often this bleeds into integration testing.
d) Our development process requires us to check in unit tests for new features and bug fixes. For bug fixes, we attempt to duplicate exactly what caused the bug and show the unit test can pass once the code fix is checked in.
e) Any of our translation layers where we translate from one data representation type to another such as POJO-XML
Discussed in SO Podcast 41 - this part is not in the transcript at the moment.
The responses in the blog contain further discussion of unit test coverage value.
Joel worries that excessive TDD (say, going from 90% to 100% test coverage) cuts time from other activities that could benefit the software product, such as better usability or additional features users are clamoring for.
Unit tests are absolutely useful as a form of “eating your own dogfood”, and documenting the behavior of your system. Even if you disregard the actual results of the tests completely, they’re still valuable as documentation and giving you a fresh outside perspective on your codebase.
Joel notes the difficulty of testing the UI (web or executable) versus testing the code behind the UI. The classic method of doing this is probably documented in The Art of UNIX Programming, where you start with a command-line app that takes in and spits out text. The GUI is simply a layer you paste on top of the command-line app, which leads to perfect testability — but perhaps not such great apps, in the long run. Which is more important?
All reasonable unit tests are valuable, but the ones that tend to save the most time in my experience, thus yielding the best ROI, are unit tests for bugfixes. Mostly because the bugfixes apply to the most difficult/brittle parts of the code.
TDD helps you write your code faster, so only TDD when going faster would provide more business value. ;)
Tests that can be reused in many places. Examples from my current project:
syntax-checking of configuration files
Checking that model properties referred to in RTF templates actually exist in the model classes (people change the classes and usually forget to adjust the templates)
Checking of some oft-overlooked style guide rules (each UI mask must have a title bar text, each button must have a unique mnemonic, etc.)
Even programmers who are skeptical about unit tests love these because they can use them with a minimal amount of work - and in some cases it leads them towards writing unit tests of their own.