First a little background. The company I work for writes web based software that is a hosted solution for our customers (ie ASP (Application Service Provider)). We are adopting agile practices such as Scrum and we execute sprints to build new features for our product.
I am a proponent of TDD (Test Driven Design), and as a part of what I deliver in a sprint I always write tests and I always get them integrated with the build (ie ccnet); however the other developers do not follow this practice and it is not enforced.
Is it a good practice to force a development group into providing unit tests as a part of what is delivered in a sprint?
Unless you are a position of authority, the best thing you can do is to convince them of the value of the test suite.
It's very difficult to get developers to see the light on this issue if they aren't seeing it done right.
Try to pair with another developer and show them the benefits and the clarity that comes from writing the tests FIRST. If you don't do this, they are likely to write all of their code, get it working, and then write tests. So, from their point of view, it will feel like simply an extra task that doesn't help them get things done.
Also keep in mind that people often do not understand how to write good tests. Even more, some do not know how to make use of tools like jmock, which can lead to them getting stuck and giving up on writing a test.
Forcing anything onto anybody is not a good practise in my view. I would show them the benefits of TDD at every opportunity I get. This should automatically get the rest of the team to voluntarily practise TDD.
You don't want lip-service unit testing, you want whole-hearted unit testing. That isn't something that can be forced. What you need to do is influence your teammates over time to see the benefits of unit testing and to develop a unit testing culture.
To start with you need to understand that different people change for different reasons. In Crossing the Chasm terms, visionaries will adopt new techniques because they are better, but pragmatists adopt new techniques either because they solve a problem/pain the currently have or because everyone else is adopting it.
Your mission then it to show how unit testing can solve a pain your team currently feels. As you win over people one-by-one eventually you can reach a tipping point where unit testing is the norm and everyone goes along with it. However if you can't tie unit testing to a pain your team feels then your efforts to convince them will likely fail.
Considering unit-testing improves code and software quality on a long term, I would say that, yes, it is good practice to have your developpers provide unit-tests -- be it part of some kind of sprint or not.
The two main barriers to unit-tests I've seen are :
the developpers don't get the point : "why would we write more code just to test ?"
"we don't have time to write unit-tests"
To answer the first point, you'll have to provide some sort of demonstration / formation, I suppose ; if you can get developpers to see why unit-tests are useful, they will like them, and use/develop them ; but they need to seen why those are useful : unless you are their boss, you cannot force people to develop unit-tests.
And, even if you are their boss, they will probably not do the best possible job if they are being forced : unit-testing is often done better if people understand why and how !
To answer the second point... Well, you obviously need to get your developpers some "special" time to developp unit-tests ; it can mean less time to do manual testing, btw.
Another thing is : it is hard to know "what to test", and "how to test" : you will need to explain / demonstrate that to your colleagues : some things cannot be tested, some things don't need to be, and some things are not "unit-testable" -- well, I suppose, unless you software is really well engineered ^^
I've gotten in that position on many jobs and contracts in the past, so I've finally gotten discouraged and embraced the darkness by advocating Development Driven Development.
I've found that when most managers embrace XP, they're embracing throwing out the documentation, not really doing TDD. Programmers on most teams are rewarded for quick hacks that get the defect out of their queue, and it's one manager in about ten who has the guts to stand up to senior management as an advocate of the overall quality of the product as opposed to the bottom-line-for-this-quarter way of doing business. After all, most software jobs that pay anything are corporate sponsored, and Freud proved that corporations are insane.
Or at least, he should have.
There is a great Joel on Software article on this called Getting Things Done When You're Only a Grunt. His strategies applied to united testing would be the following:
Just Do It
Most important, I think. Regularly write tests, do not make it appear as if you yourself see them as a minor part of development.
Harness the Power of Viral Marketing
If you see an error in someone else's code, write a test that triggers it and present both to him. Maybe he sees your point.
Create a Pocket of Excellence
Identify the team members who are open to the idea but not quite sure how to work with unit tests and set up a number of test classes until everyone of them knows how to do it. Then turn towards the other ones. Things are a lot easier to establish once you are not the only one anymore.
Neutralize The Bozos
There will be team members who are almost impossible to get to write tests. Maybe those should be dealt with by regularly breaking their code with a new commit - and then pointing out that since they don't have tests it was hard for you to notice.
it depends on your Version Control, but there are Version Control Software that enable you to run scripts before check-in or before merge to the Production branch. and it will not let the developer check in if the unit testing is failed.
First, talk to your manager. If he is convinced that testing is a good thing, add a coverage test to your build system. If the coverage of your unit tests falls below a certain level, you can handle it as a failed build. This gives your colleagues a measure, a way to see when the fail to deliver tested code.
In your case, NCover seems to integrate nicely into CC.NET.
Create a matrix that you want developers to achieve.
Make sure there is a subtask for Unit test creation for every story.
No. In my experience, TDD isn't all that useful in practice. I sometimes use it for really general fundamental classes (like geometry or generic data structures) that lend themselves naturally to automated tests. But for UI components or business logic, I find it's more trouble than it's worth.
Related
So - management is looking to do a push to move towards doing unit-testing in all the applications moving forward - and eventually get into full TDD/Continuous Integration/Automated build mode (I hope). At this point however we are just concerned about getting everyone developing apps moving forward using unit-testing. I'd like to just start with the basics.
I won't lie - I'm by far no expert by any means in unit-testing, but I do have a good enough understanding to start the initiative with the basics, and allow us to grow togeather as a team. I'd really love to get some comments & critisism from all you experts on my plan of attack on this thing. It's a team of about 10 developers in a small shop, which makes for a great opportunity to move forward with agile development methodologies and best practices.
First off - the team consists of mainly mid level developers with a couple of junior devs and one senior, all with minimal to no exposure to unit testing. The training will be a semi-monthly meeting for about 30-60 minutes each time (probably wind up running an hour long i'd guess, and maybe have them more often). We will continue these meetings until it makes sense to stop them to allow others to catch up with their own 'homework' and experience - but the push will always be on.
Anyway - here is my lesson plan I have come up with. Well, the first two at least. Any advice from your experts out there on the actual content or structure of the lessons, etc, would be great. Comments & Critisism greatly appreciated. Thanks very much.
I apologize if this is 'too much' to post in here or read through. I think this would be a great thread for SO users looking to get into unit testing in the first place as well. Perhaps you could just skip to the 'lesson plans' section - thanks again everyone.
CLIFF NOTES - I realize this post is incredibly long and ugly, so here is the cliff notes - Lesson 1 will be 'hello world unit tests' - Lesson 2 will be opening the solution to my most recent application, and showing how to apply each 'hello world' example in real life... Thanks so much everyone for the feedback you've given me so far.. just wantd to highlight the fact that lesson 2 is going to have real life production unit tests in it, since many suggested I do that when it was my plan from the begining =)
Unit Testing Lesson Plan
Overview
Why unit test? Seems like a bunch of extra work - so why do it?
• Become the master of your own destiny. Most of our users do not do true UATs, and unfortunately tend to do their testing once in production. With unit-tests, we greatly decrease risk associated with this, especially when we create enough test data and take into account as many top level inputs as we possibly can. While not being a ‘silver bullet’ that prevents all bugs – it is your first line of defense – a huge front line, comparable to that of the SB championship Giants.
• Unit-Testing enforces good design and architecture practices. It is ‘the violent psychopath who maintains your code and knows where you live’ so to say. You simply can’t write poor quality code that is well unit-tested
• How many times have you not refactored smelly code because you were too scared of breaking something? Automated testing remove this fear, makes refactoring much easier, in turn making code more readable and easier to maintain.
• Bottom line – maintenance becomes much easier and cheaper. The time spent writing unit tests might be costly now – but the time it saves you down the road has been proven time and time again to be far more valuable. This is the #1 reason to automate testing your code. It gives us confidence that allows us to take on more ambitious changes to systems that we might have otherwise had to reduce requirements on, or maybe even not take on at all.
Terminology Review
• Unit testing - testing the lowest level, single unit of work. E.G. – test all possible code paths that a single function can flow through.
• Integration testing - testing how your units work together. E.g. – run a ‘job’ (or series of function calls) that does a bunch of work, with known inputs - and then query the database at the end and assert the values are what you expect from those known inputs (instead of having to eye-ball a grid on a web page somewhere, e.g. doing a functional test).
• Fakes – a fake is a type of object whose purpose is to use for your testing. It allows you too easily not test code that you do not want to test. Instead of having to call code that you do not want – like a database call – you use a fake object to ‘fake’ that DB call and perhaps read the data from an XML/Excel file or a mocking framework.
o Mock – a type of fake to which you make assert statements against.
o Stub – a type of fake to which you use as placeholder code, so you can skip the database call, but do not make asserts against
Lessons
Lesson one – Hello Worlds
• Hello World unit test - I will create a ‘hello world’ console application that is unit tested. Will create this application on the fly during the meeting, showing the tools in Visual Studio 2008 (test-project, test tools toolbar, etc.) that we are going to use along the way while explaining what they do. This will have only a single unit-test. (OK, maybe I won’t create it ‘on the fly’ =), have to think about it more). Will also explain the Assert class and its purpose and the general flow of a unit-test.
• Hello World, a bit more complicated. Now our function has different paths/logical branches the code can flow through. We will have ~3 unit tests for this function. This will be a pre-written solution I make before the meeting.
• Hello World, dependency injection. (Not using any DI frameworks). A pre-written solution that builds off the previous one, this time using dependency injection. Will explain what DI is and show a sample of how it works.
• Hello World, Mock Objects. A pre-written solution that builds off the previous one, this time using our newly added DI code to inject a mock object into our class to show how mocking works. Will use NMock2.0 as this is the only one I have exposure to. Very simple example to just display the use of mock objects. (Perhaps put this one in a separate lesson?).
• Hello World, (non-automated) Integration Test. Building off the previous solution, we create an integration test so show how to test 2 functions together, or entire class together (perhaps put this one in a separate lesson?)
Lesson two – now we know the basics – how to apply this in real life?
• General overview of best practices
o Rule #1- Single Responsibility Principal. Single Responsibility Principal. Single Responsibility Principal. Facetiously stating that this is the single most important thing to keep in mind while writing your code. A class should have only one purpose; a function should do only one thing. The key word here is ‘unit’ test – and the SRP will keep your classes & functions encapsulated into units.
o Dependency Injection is your second best friend. DI allows you to ‘plug’ behavior into classes you have, at run time. Among other things, this is how we use mocking frameworks to make our bigger classes more easily testable.
o Always think ‘how will I test this’ as you are writing code. If it seems ‘too hard to test’, it is likely that your code is too complicated. Re-factor until it into more logical units of classes/functions - take that one class that does 5 things and turn it into 5 classes, one which calls the other 4. Now your code will be much easier to test – and also easier to read and refactor as well.
o Test behavior, not implementation. In nutshell, this means that we can for the most part test only the Public functions on our classes. We don’t care about testing the private ones (implementation), because the public ones (behavior) are what our calling code uses. For example... I’m a millionaire software developer and go to the Aston Martin dealership to buy myself a brand new DB9. The sales guy tells me that it can do 0-60 in 3 seconds. How would you test this? Would you lift the engine out and perform diagnostics tests, etc..? No... You would take it onto the parkway and do 160 MPH =). This is testing behavior vs. implementation.
• Reviewing a real life unit-tested application. Here we will go over each of the above ‘hello world’ examples – but the real life versions, using my most recent project as an example. I'll open a simple unit test, a more complex one, one that uses DI, and one that uses Mocks (probably coupled to the DI one). This project is fairly small & simple so it is really a perfect fit. This will also include testing the DAL and how to setup a test database to run these tests against.
I like the commitment and thoroughness you're expressing here, but my feeling is your adoption plan is going to take longer this way than if you started straight in, paired up and wrote some actual production tests. You'll need infrastructure anyway -- why not bootstrap your actual production code with a CI server and a TDD framework? (A lot of times this is a bigger pain point than learning "asserts." Get this part out of the way sooner rather than later). Then sit down and solve an actual problem with a fellow coder. Pick a simple bug or small feature and try to identify what the failing tests would look like and try to write them.
I like Hello Worlds for a brown bag lunch or something. But I can't think of any reason not to just jump right in and solve some problems.
I helped design and run a Test Driven Development training in house at my company for Java developers. We ended up doing it as a full day training, and we broke it up similar to how you have here.
Why do testing?
Simple example.
More complex example.
One thing I must stress though, is people need to learn by doing.
For my training, we set up a lab environment where each student could start with the same snapshot of code, and develop and run the tests themselves, with instructors walking around the training room helping individuals who were confused or weren't getting it.
For the "Simple Example" we had a "cooking show" version of code that was on a projector and went through the TDD process step by step. Developers would have to go through the process of writing a test, then creating implementation that was just sufficient to pass the test, then repeat. At each phase, a prepared solution to the current phase was shown on the projector after the students had some time to try it on their own.
For the "Complex Example" we created a set of requirements, and then allowed the students to come up with their own solutions using TDD to do so. We even had an exercise where requirements surprise, surprise changed suddenly part way through the exercise.
I like your idea of doing it over a longer period of time with regular checkups. One downfall of our training was a lack of followup. Developers benefited by the training, I'm sure, but many I think, did not take the practice back to their ordinary work. Regular checkups would help instill unit testing as a matter of habit.
For more ideas, check out my answer to this question.
Getting people interested in unit testing on someone else's/sample code will not be as effective as having folks "start with the testcase" on some new code they have to write.
Cover the basics of starting with a test case and triangulating to a valid result (all the while emphasizing that a failing test is progress).
That said, my personal experience is that things like this are better done in a pair programming environment. It's extra work for you, but the combination of one-on-one, some ownership and a working example they can reference at the end of it will make adoption much easier.
Getting a coverage tool running at some point later on can, given the right environment, provide a fun, competitive and gratifying way of marking progress. Making this in any way part of compensation/bonuses is a real bad idea. Done correctly, folks will adopt it because it works.
I tried to introduce TDD in a small team (7-9 programmers). Lectures failed. People are skeptical, TDD sounds like snake oil. Plus, there's always the proverbial Willy.
In the end, instead of TDD, people on the team do occasional DDT (Development-Driven Tests): the write unit tests after the code to confirm that it does what they meant it to. Looks like utter failure until you realize that even this form of developer-driven QA is much better than what you had before because, even if it doesn't really improve the code for the future revisits, it cuts into deployed bugs.
There was, however, one key difference compared to your setting: the push didn't come from the management, they were just good enough to let the programmers decide for themselves.
The lectures failed, but I didn't put all the eggs in a single basket. I wrote a bunch of tests for many of the smaller, more focused functions and classes used across the project. Those little buggers you occasionally need to bend a bit to fit exactly in the next call site, hoping you didn't break any of the existing ones (you looked around the code, the change seemed safe). It was running the tests after every other svn up (not only by me), and the subsequent replies to the guilty commit emails "you broke it, pls fix ASAP!" that finally convinced them of at least partial utility of unit tests.
No matter what examples you come up with for the lectures, they'll tell you it only worked because the CUT was unrealistically (or unusually) isolated from the rest of the code. They'll ask you to write unit tests for a convoluted, buggy God Class (preferably a Singleton, everybody loves global variables, no?) because, to the best of their knowledge, that's what real-world code looks like, and they think it'll look like that with TDD as well.
Screw lectures. Have the management buy everybody a few books for the Christmas: at the very least TDD By Example (Beck) and Refactoring (Fowler). Speaking from experience, the books won't necessarily have any impact with most of them (more of the "that wasn't real-world enough" bullshit), but it's bound to stick to someone, and, no offense intended, I'd bet $1000 that Beck is the better evangelist of you two. ;) Practice TDD yourself in code you share with them. If it really works, they'll follow. Sss-looowww-lyyyy, but they will.
Then again, maybe you don't have that many Willies on your team, maybe your colleagues are eager, just in need of a leader, I don't really know, you didn't tell.
First commandment: don't be put off by slow and/or partial intake: every inch counts! (I'm preaching to myself here, ya know...)
I think it will be harder to start with test-after type of testing. A major difference compared to test-first is that test-after does not guide you in writing code which is easy to test. Unless you have first done TDD, it will be very hard to write code for which it's easy to write tests.
I recommend starting with TDD/BDD and after getting the hang of that, continue learning how to do test-after on legacy code (also PDF). IMO, doing test-after well is much harder than test-first.
In order to learn TDD, I recommend finishing this tutorial. That should get you started on what kind of tests to write.
I think it's a good outline for a session or two. But I'd strongly recommend standing up a testing infrastructure first and setting the developers up for early success, rather than shoving the developers head-first into the unit testing world and shouting "ready, set, GO!"
Assuming you develop in an IDE, a one-click test generator tool will really help with step one, test creation. You'll want to integrate a GUI-based test runner with your IDE, and you really need an automated test runner integrated with your CI builds. Having a code coverage tool integrated with the test runner tool will help developers increase their coverage. You need to provide developers with a complete framework for writing the tests, and that includes documenting how to organize the test source code (test naming standards, project settings, folder locations, etc.) Tools that automate any of the manual steps will go a long way. Provide a mock framework including mock objects for any in-house or third-party library APIs you depend on. If you're going to recommend a database full of fake data (instead of a suite of fake data access objects) create one with a few key tables and populate it with test data, and provide whatever it takes to reset it between testing runs. If not, provide a few fake DAOs of your own to serve as examples. You should have a small suite of tests already in your source control system testing production code to show as examples. And you need to have all this documented so you can hand it out in meeting #1. (Don't forget to test the document!)
One of our teams tried the ad hoc approach a couple years ago, but adoption was poor. Without the definitions or the organization of the tests, individual developers kind of did their own thing. Lack of a naming standard made it difficult to identify the right tests to run for which modules, and few people bothered. We had no IDE integration for any of it. Nobody wanted to write mocks, and we didn't provide mocks in advance for the framework we use. And it's really hard to add tests to legacy code that wasn't written with Dependency Injection in mind.
I'm now trying to get our infrastructure reorganized and ready for another shot at the task. Only after it's standing up will I work on trying to get the developers to use it again.
However, you have the one thing we were kind of lacking last time, and that's strong management support. Our previous managers were kind of lukewarm to the idea because they didn't want to delay coding by having to write all these tests. We now have new management and they are focusing on code quality -- automated unit tests are now in vogue, so it's a good time for us to try again.
Is a classroom style approach going to work well? That's a question that comes to my mind as what may happen here is so much of a separation between knowledge and application of the ideas that while they may know it, they don't quite get how to use it.
Just to give a different approach, one could jump into whatever projects are going on and start trying to apply some of this which may take a while to trickle down through everyone as the idea would be that you'd train one and then as a pair you'd train another pair and so on, to try to get everyone up to the same point. This way the theoretical stuff that isn't relevant gets taken out early.
Another idea would be to see if it makes sense to try to form teams within the group of 10, such as 2 groups of 3 and a group of 4 or 5 pairs, so that each person can present what they've done and how well do they use this.
I'd second the boundary testing, as well as showing why it can be useful to test with invalid input to see that this doesn't cause the system to roll over and die.
Lastly, it may make sense to have time devoted to handling questions and other 1:1 issues that people have as not everyone will want to ask questions in front of a large group.
EDIT: A caution on Lesson 2, which is a good idea but has a few hazards to note. First, be humble in doing this lesson so that you aren't coming across as a show-off wanting to slam everyone else. Second, I'd suggest leaving some tests incomplete when going over things so that there is an interactive aspect here that prevents the other developers from merely tuning out or rolling their eyes. Third, a follow-up lesson or two of others showing what they did or you going over what they did may be more useful to show that this is a team effort in some ways.
If management is serious about this, then they have to take the bull by the horns and take action themselves not just have a little training. Start having them ask to see the tests in code reviews. Once this is common and people know they willnot pass the code reveiw without a test, then have them refuse to deploy new code until there are tests written. It will only take a couple of iterations of this before people will know that they can't get away wihtout writing the tests and then they will do so fairly reliably. Yes the first time the manager refuses to deploy, something will probably be late. They have to take that into accound in their own planning. They also will need to start adding test writing time to their task estimating. No one wants to do this if they think they will be blamed because it takes longer to write tests and code than just to write code. Have a published schedule for how this will be implemented (you need to give them some time to learn how to do this before it is required for all new development) and on the drop dead date, nothing gets to prod without tests.
I like it. One comment I have is it's probably worth dropping in some guidelines/advice on how to go about testing threaded/asynchronous code.
Also, I'd make a comment about boundary testing i.e. using unit tests to state your assumptions on any 3rd party API you utilize.
Just because I have been doing something where TDD (and unit tests in general) has been invaluable and because it might make a good demo, I'd suggest an example using Regex, especially if you aren't a Regex expert, it's quite easy to make mistakes.
It'd also make for a good 'show, not tell' demo. If you just:
step 1: Tell them just to write some code with a Regex to match a certain pattern.
step 2: expand the pattern to something else.
step 3: show how unit tests immediately let them be sure that a) it matches the new pattern and b) it hasn't broken any of the previous matches or let through results that shouldn't be.
In general C&C of your review, it's hard to get an idea of exactly what sequence events will go in. Also, semi-month seems far too infrequently if there's just a void inbetween with no push to apply their knowledge. The last thing you want is it to become some meeting they go to and forget about as soon as they are out of the room!
Do you have sufficient downtime in your current workload to address technical debt and get the team to actually pair up and start looking at adding unit tests to some existing functionality and refactoring where necessary to allow mocking?
I would say that will prove far more effective than lectures and potted workshops, YMMV.
A new project we began introduced a lot of new technologies we weren't so familiar with, and an architecture that we don't have a lot of practice in. In other words, the interfaces and interactions between service classes etc of what we're building are fairly volatile, even more so due to internal and customer feedback. Though I've always been frustrated by the ever-moving specification, I see this to some degree a necessary part of building something we've never built before - if we just stuck to the original design and scope, the end product would probably be a whole lot less innovative and useful than it's becoming.
I also introduced test-driven development (TDD), as the benefits are well-documented and conceptually I loved the idea. Two more new things to learn - NUnit and mocking - but seeing all those green circles made it all worthwhile.
Over time, however, those constant changes in design seemed to mean I was spending a whole lot more time changing my tests than I was on writing the code itself. For this reason alone, I've gone back to the old ways of testing - that is, not automated.
While I have no doubt that the application would be far more robust with hundreds of excellent unit tests, I've found the trade-off of time to launch the product to be mostly unacceptable. My question is, then - have any of you also found that TDD can be a hassle if you're prototyping something / building a beta version? Does TDD go much more naturally hand-in-hand with something where the specifications are more fixed, or where the developers have more experience in the language and technologies? Or have I done something fundamentally wrong?
Note that I'm not trying to criticise TDD here - just I'm not sure it's always the best fit for all situations.
The short answer is that TDD is very valuable for beta versions, but may be less so for prototyping.
I think it is very important to distinguish between beta versions and prototyping.
A beta version is essentially a production version that is just still in development, so you should definitely use TDD in that scenario.
A prototype/proof of concept is something you build with the express intent of throwing it away once you've gotten the answers out of it that you wanted.
It's true that project managers will tend to push for the prototype to be used as a basis for production code, but it is very important to resist that. If you know that's not possible, treat the prototype code as you would your production code, because you know it is going to become your production code in the future - and that means you should use TDD with it as well.
When you are learning a new technology, most code samples etc. are not written with unit tests in mind, so it can be difficult to translate the new technology to the unit testing mindset. It most definitely feels like a lot of overhead.
In my experience, however, unit testing often really forces you to push the boundaries of the new technology that you are learning. Very often, you need to research and learn all the different hooks the new technology provides, because you need to be able to isolate the technology via DI or the like.
Instead of only following the beaten path, unit testing frequently forces you to learn the technology in much more depth, so what may feel like overhead is actually just a more in-depth prototype - one that is often more valuable, because it covers more ground.
Personally, I think unit testing a new technology is a great learning tool.
The symptoms you seem to experience regarding test maintainability is a bit orthogonal, I think. Your tests may be Overspecified, which is something that can happen just as well when working with known technologies (but I think it is probably easier to fall into this trap when you are also learning a new technology at the same time).
The book xUnit Test Patterns describes the Overspecified Test antipattern and provides a lot of guidance and patterns that can help you write more maintainable tests.
I've found that thoroughly testing early results in lots of code thrown away and an empty feeling in the pit of your stomach.
Test what needs to be tested and not a line of code more. When you figure out how much that is, let me know.
When prototyping, I would say it depends on the type of prototyping. In evolutionary prototyping, where the prototype evolves into the final application, I would utilize unit testing as early as possible. If you are using throw-away prototyping, I wouldn't bother with unit testing - the final application is going to be nothing like the prototype.
I'm not sure what you mean by "beta", since a beta is almost a finished product. As soon as you start working on code that is going to be shipped, or has a potential to be shipped, make sure everything is well tested.
Now, pure test-driven development might be extreme, but it is important to make sure that all shippable code is as tested as possible, at the unit, integration, and system level.
have any of you also found that TDD can be a hassle if you're prototyping something / building a beta version?
I have.. Tons of times :)
Does TDD go much more naturally hand-in-hand with something where the specifications are more fixed, or where the developers have more experience in the language and technologies?
Not really. TDD works quite nice with changing requirements, but TDD is really for ensuring a stable and contract-driven design: Two things which prototypes doesn't really need that badly..
Or have I done something fundamentally wrong?
Doesn't look like it :) You've just seen that TDD consists of other things than golden trees..
"Over time, however, those constant changes in design seemed to mean I was spending a whole lot more time changing my tests than I was on writing the code itself"
Good. You should spend a lot of time on testing. It's important, and it's how you demonstrate that your code is right. "Less code than test" is a good benchmark.
That means that you were writing effective tests that demonstrated your expectations for the underlying technology.
You may want to consider doing this.
Some tests are "essential" or "core" or "enduring" features of the application independent of any technology choices. Focus on these. The should never change.
Some tests confirm the technology or implementation choices. These change all the time. Perhaps you should segregate these so that the technology changes lead to focused changes here.
Being given a roadmap with a moving X is frustrating.
TDD or no TDD.. 'having to spend majority of the time changing the tests instead of the code' indicates either that the specs were changed radically or you just over-mocked yourself a.k.a "fragile tests". I'd need more input from you to comment further.
Spike/Prototyping means trying to build something as a proof of concept or validation a design. With this definition, I'd say that you don't need to TDD your proto because your primary goal is learning / reducing the unknowns. Once you've done that you should throw away your proto and build your production version with automated tests (use TDD now). You ship these to the customer not 'tested prototypes'
However if manual testing has been working well for you, persist with it. I like to prove to myself and others that my code works at the push of a button - avoid human boredom of repetitive tasks and get thorough testing.
Shipping Prototypes will bite you sooner and harder than you ever imagine. Take it from me.
Prototyping is meant to be the used for "Would this kind of thing work"-exploration.
So there is no need for Testing. BUT! Always throw your Prototype away and code from ground zero!
In agile development there's the concept of a "spike" - a deep but narrow investigation of a solution or a technology. Once you're comfortable with how things should work you start over with a new implementation with a higher quailty level.
There's a pitfall with software labeled as "beta" - all of the sudden you end up with something not intended for production as a vital part of your application and you haven't got the time to redo it. This will most likely come back and bite you later on. A protoype should really be just a prototype - no more, no less.
I don't have a prescription for you, but I do have a diagnosis:
if you ever end up in a debugger, you shoulda had more tests. Even in the very earliest prototypes, you want the code you write to work, right? If I don't do TDD, and I run in to a bug, it's hard to retrofit unit tests to find the bug. It's tempting to go to the debugger. So, I aim my TDD efforts to produce a good enough set of tests so that I don't need the debugger. Doing this well requires lots of practice.
If you want to refactor but don't because of risk, you shoulda had more tests. If I'm going to be working with some code for more than a couple hours, I want to refactor to keep my velocity high. If I hesitate to refactor, it's gonna hurt my productivity, so I do everything I can to make refactoring safe and easy. Again, a good selection of tests is exactly what I need.
For pure prototyping, as people have said, not so useful. But often a prototype has elements of the application that will follow.
As you build it, what are the solid concepts that you expect to continue? What are the key domain models? It makes sense to build tests around these. Then, they provide a solid base on which to explore ideas. By having part of the code base solid and tested, it allows you to go further with the prototype than you could with no tests.
It's always a balance of picking the right things to test when. It sounds like from your description you are adopting quite a few new things at once-- sometimes this can be too much, and you need to retreat a little bit to optimize your progress.
What I typically do for this prototyping code is to write it without tests (or not many), but do the work under my src/test/java or wherever I'm putting my test code. That way, I won't inadvertently put it into production. If there's something I've prototyped that I like, then I'll create something in src/main/java (my prod code) and TDD it, pulling over code from the prototype one piece at a time as I add tests.
Over time, however, those constant changes in design seemed to mean I was spending a whole lot more time changing my tests than I was on writing the code itself.
I write (and run) automated tests of the system's I/O. The system's I/O depends on the functional requirements, and doesn't change when the system's implementation changes, so I can refactor the implementation without changing the automated tests: Should one test internal implementation, or only test public behaviour?
When I get in contact with a new technology, I usually write a couple of tests which I use as the basis of the prototype. The main advantage here is that it allows me to get used to the technology in small, digestible parts and that I document my progress in the only valid form of documentation: code.
That way, I can test assumptions (what happens when I persist an empty string in a database? DB2 will return an empty string, Oracle will return NULL) and make sure they don't change while I work on the prototype.
Think of this work as many, tiny prototypes all rolled into a couple of unit tests.
Now the design of my product will change over time but these spike tests are still valid: The underlying framework doesn't change just because I now pursue a different solution. In fact, these tests will allow me to migrate to the next version of the technology since they will ring an alarm when one of the assumptions I've been using has changed. Otherwise, I'd have to stay with the current version or be very careful with any upgrade since I couldn't be sure what it would break.
I would be cautious about dismissing TDD when prototyping. Primarily because prototypes tend to (but not always) evolve into the final product. If you can be sure that the prototypes are thrown out once they have established what ever it was they were started for, then you can be a bit more lax. If however it there is a chance that either the prototype will evolve into the final product, or parts of the code will be ported into the final product, then would endeavor to follow TDD principles.
Not only does this make your code more testable, but more importantly (in my mind) it encourages you to follow good software design principles.
If it's actually a prototype and you are going to throw it away when you're done, then take whatever route gets you through that phase. But, ask yourself, how often do your prototypes actually get thrown away vs. how often do they creep into you're final product?
There was a good presentation at JavaOne on TDD. One of the really nice outcomes was that you tended to understand your actual requirements much, much better when you had to author tests to meet them. This is in addition to all of the refactoring and code quality benefits that came along with the approach.
We have a group of a few developers and some business analysts. We as developers would like to start adding unit testing as part of our coding practices so that we can deliver maintainable and extensible code, especially since we will also be the ones supporting and enhancing the application in the future. But in this economic downturn we are struggling with the push to get started because we are challenged to just deliver solutions as fast as possible, with quality not being the top priority. What can we do or say to show that we will be able to deliver faster and with higher quality, as well as preparing for future enhancements.
Basically we just need to get over the learning curve of incorporating unit testing into our daily work, but we cannot do that now because it is viewed as an unnecessary overhead that would delay our projects that the business needs now.
We as developers want to provide the highest value to the business, especially quickly, but we know that we will also need to do this 6 months from now and we need to plan for that as well, and we believe that unit testing will help us greatly down the line.
EDIT
All awesome input, thank you. I personally know how to write unit test, but I don't have the experience in me to say whether or not that unit test is good. I have just ordered Test Driven Development: By Example and will take the initiative to get the ball rolling on incorporating unit testing in our group.
You need to just start doing it, with or without permission. In the end it will make you more productive and increase your code quality. You can start small by including units for something critical and once you've shown the benefits, you're in.
Start unit testing the functions or classes when you create them. Begin with simple classes/functions that do not have external dependencies (DB, file system).
Share your progress inside the team. Count the number of tests and display a big chart showing your progress (unless the management/analysts are very hostile against unit testing).
Read about TDD: "Test-Driven Development : by example". Writing the tests first leads to code that can be easily tested. If you write the production code first, you may have hard time putting it under tests.
I would like to recommend the book
Pragmatic Software Testing
The book Pragmatic Unit Testing is also a good book, although it focuses more on C# and NUnit, there are topics which are applicable language independent
Just do it...
If it's part of your personal process for any new code, you'll at least have that covered. You'll probably never get permission to add unit tests to cover all your code ever but you can at least be sure that no future changes undo work from a given point in time.
Think of it from the business side, why do you need to write code to prove the code you've already written is correct? Why is it wrong?
Getting 100% coverage would be nice but take your time getting there and don't write tests for existing code that isn't currently wrong; write the tests as it breaks so you at least never undo something unintentionally.
You don't even need to discuss it with management (though the situation you describe about it is far from ideal). It's like Design by Contract - I introduced it in previous code I worked in, just as part of the development process. Once it's in, and it works, trust me, noone will dare to remove it.
Unit testing can also be seen as a part of development. Hence, if you have "20 days" for developing features A, B and C, you can typically include unit tests in your estimations for development itself.
Making unit tests is surprisingly easy. Compared with multithreading problems or any sort of complex design, unit testing is very easy to grasp for any competent developer.
You can read good literature about it (you have dozens of tutorials online) in, say, half a day, and start doing your first tests in the afternoon.
Really - just do it!
I know this may not be readily available to you, but I believe getting someone who's test infected on your team to show you the way is the easiest way to maintain productivity when introducing testing. Your people obviously need to know the concepts, which they can do by reading books or attending user groups. A test infected developer can show you how to do it in your practical everyday work with an absolute minimal loss in productivity whilst doing so. It wouldn't surprise me if your speed increased immediately, but I wouldn't make any claims of that sort. With test experience also comes knowledge of testable designs, which I think is key.
Does anyone have metrics on the utility of formal Unit Testing? I see a lot of attention being paid to unit testing tools and I was curious why?
I stopped formal unit testing over 5 or 6 years ago and the net gain in productivity seems quite high. I stopped unit testing because I noticed that it never caught anything - let alone anything useful. The type of errors that unit testing detects seem to be preventable by not drinking more than 2 glasses of wine/beer per hour (or 2 joints per hour). Also - unit testing seems to create risk by allowing the developer to think that there is some safeguard to catch their mistakes.
I do test to ensure that the code works as it should, but I do not use any tools. I test based on the changes being made. My production error rate for new code is approximately zero. My error rate for changes to code is about 2-3 bugs per quarter. The above measures are based on 4 production applications that I develop/support.
I acknowledge your superiority as human being and a coder.
I, however, am a mere moron, and without Python unittest, I would be lost.
I cannot refactor without unit tests, it just takes too much thinking.
I can barely code without unit tests, it's too hard to be absolutely sure I absolutely understand absolutely every nuance.
I unit test because I'm an idiot. Since you don't make mistakes, you clearly don't need to unit test. I salute you.
Edit. For me, unit tests aren't about metrics or costs. I don't need any randomized, controlled experiments to show me the value. I cannot work without them. Indeed, I refuse to work without them. In a similar vein, I won't work without a compiler, a text editor, or source code control; I won't work without requirements; I refuse to program without doing design first.
I do not see unit testing as a replacement for traditional testing, but rather as an extra step to ensure correctness of code. Some particular areas where I find unit testing useful are:
When refactoring/changing existing code. Unit tests will verify that at least those cases still work as expected. The more tests you have, the more sure you can be that the code changes did not break existing code.
When submitting bug reports. Having a unit test which exposes a bug is a great way of demonstrating the bug AND knowing when it has been fixed.
A means of designing interfaces. You have some test code to check the interfaces out with.
Probably a few others I've forgotten about :-P
PS: How do you know you make no bugs? I don't think that I introduce bugs into code I work on, but that certainly doesn't make it so. IMHO, it is naive to think that your code is bug free.
(Regarding unit testing, if you know your code may contain bugs - and I would say most code does - wouldn't you want to use every possible means to catch them?)
Here is some White Paper about Unit Test that might help you:
White paper #1
White paper #2
White paper #3
But, Martin Fowler put it, the anecdotal evidence in support of unit tests and TDD is overwhelming, but you cannot measure productivity.
Unit testing is good because you can change a part and know if somewhere else it has modified something. Some people are "in love" with Unit Testing and should calm theirselve. I believe in Unit Testing but people who try to covert everything are AS dangerous of people who do not unit test.
Here is a thread that has some research about the TDD approach
Research on TDD
Is there hard evidence of the ROI of unit testing?
I don't have any metrics to point at, but I think the rise in popularity is because the rest of us have had experience that's the opposite of yours. :)
With unit tests, I can fix bugs in production code and install the new version within the hour the bug was found and I can be sure that the new version isn't worse than what we had before - because the tests tell me so. It might be better, though.
They give me a lower watermark below which the quality of my code can never sink. They allow me to keep track of the bigger picture and have the tests find the small mistakes that I tend to make. They also allow me to develop in a very relaxed style.
Since I test, I tend to deliver on time, my code quality has improved a lot and the result usually works as expected. Also, I'm much faster since I can cut corners which would be too dangerous to try if I didn't have the tests.
That said, I also don't have any hard numbers nor do I know any source despite the fact that I'm doing unit test and TDD for years. My love for tests is based on pure word of mouth and personal experience.
I've found that unit testing helps me when adding new functionality. In this scenario I used to worry that what I was adding was going to break something in some remote part of the application. But with appropriate unit tests I know whether or not I've broken something the moment I run the tests.
Here's an interesting discussion on the utility of unit tests.
If you don't like unit tests, another concept you might want to look into is Design By Contract. Which basically asserts that if certain input conditions are met then there will be a guaranteed output according to the contract.
I'm a development manager. For my organization, setting up and migrating to nhibernate involved some setup costs and added to our development time. Some of the developers liked it, some thought it was a waste of time.
There hasn't been a noticeable change in error rates, but perhaps it's too early to tell.
From my perspective, I think it helps junior developers who aren't sure of their work, but for the senior developers, it seems to slow them down - it's one more thing to keep updated. I'm not sure if we'll continue using this, revert back to our old ways (ad hoc unit testing), or let developers make a personal choice.
There are several tools that measure the code coverage with unit tests.
They are an essential part together with unit test to ensure the code is not only tested, but completly tested.
Everything else is just pure magic ;)
If you want to refactor code, by definition you need some way of telling if the changes broke the code. Lacking divine insight, I find that unit testing is a pretty good tool, but ymmv.
I've specifically had a lot of gain using Test Driven Development (TDD) with C++ on a huge monolithic server application.
When I'm working on an area of code, I first ensure that that area is covered with tests before I change or write new code.
In this use case I have huge gains in productivity.
To build and run my test: 10 seconds.
To build, run and manually test the full server: min 5 minutes.
This means I'm able to iterate an area of code very quickly and only test fully when I need to.
In addition to this I have also utilised integration testing which take longer to build and run, but only exercise the specific piece of functionality I am interested in.
I have some sympathy for what you're saying. My experience is similar in that my new bugs are seldom caught by unit tests. If at all, the unit tests are modified after the bug has been found to ensure that it doesn't reappear.
Where unit tests have helped me is in the design of my (Java) classes. I have often refactored the classes to make them testable (removal of singletons for example) which, I think, has improved the overall design. For me, that's reason enough to use them.
In my experience Unit Testing helped me with the following stuff :
Now I can give all of my focus to the code block / function / class that I'm writing without worrying about anything else, because I know If I do something stupid or cause a side effect my tests will tell me
I can refactor stuff by knowing that I'm not breaking stuff
I'm sure that my app is working as expected
Before a release I don't check every single functionality manually to confirm that app still works,
I'm sure my application is always stable in some level
However this is a bit related with me since I'm really bad about "managing multiple stuff at a time" and "focusing". Therefore Unit Testing works for me and literally save my day so many times, where I introduced a new feature and it broke some old functionality.
If this is not the case for you just don't use it. If you are still having the same outcome, performance and quality with the same amount of bugs then that means you don't need unit testing. Or you need to revise your unit testing methodologies.
P.S. Based on your bug rate and what you've said you sound like a genius anyway (assuming these are medium or big projects), so I'd say don't use Unit Tests, it looks like you are doing fine. But for the rest of the world who are not genius such as me I strongly recommend Unit Test because it worked for me.
Dunno about you, but I've just checked in two fixes for two coredumps created by changes in existing code that my unit tests caught. And I just had a major production issue that would have been caught by a unit test if I had more trust in its results (our unittests are a bit more on a functional side than I'd like to admit).
It seems to me that formal Unit Testing with the popular testing tools is pretty much like U.S. airport security.
It provides the illusion of security
It makes people 'feel' good
It's very inconvenient (extremely inconvenient if you got the wrong color skin)
People will angrily wave their fist at you if you criticise it...
These same people will be left befuddled when their process fails them and they will jump on the next band wagon...
I think people have different perspectives on software. In my opinion, software is a means of making money (hopefully by providing increased revenues or saving money). I've seen the posts for TDD which is the closest I see as a scientific way to answer the question, but the methodology lacks scientific rigor. None of the articles specified had a baseline or fairly contrasted alternative method.
I guess, the fans of formal unit testing will continue to feel secure in their ways. I will continue to write my spec on a scrap of paper and put in the bowl at the feet of my statue of St. Anthony and say a prayer. Who is to say which way is more effective, but my way sure feels good... Maybe I'll write a white paper about it.
Remember the rise in popularity of 70's and 80's haircuts and clothes... that didn't work out so well for those of us who lived in those decades.
Formal unit testing takes considerable work and effort to maintain. I'd guess that it takes 20-50% of the time it takes to actually develop the software. What I'm asking is for the known price of adding 20-50% overhead to every development effort, is the gain noteworthy and/or proveable.
By not doing formal unit testing, you are forcing the developer to think through the appropriate things to test. The developer takes more ownership of the product.
Formal Unit testing sounds like snake oil juice... Everyone and his brother say it is good, useful, cool, etc., but there has not been a randomized controlled trial to prove that it actually saves time or money. All the responses thus far are subjective testimonies.
What I'd like to know is if there is a software manager who can demonstrate higher productivity (or even higher quality) after the introduction of unit testing.
lol - the facetious rant by S.Lott is the highest ranked response... Given this forum is anonymous (for me anyway), your respect is not what I'm seeking. I'd consider myself barely above mediocre. I've worked with exceptional developers... those guys generally don't even do basic tests of their code - they just know it will work.
This certainly presupposes that unit testing is a good thing. Our projects have some level of unit testing, but it's inconsistent at best.
What are the most convincing ways that you have used or have had used with you to convince everyone that formalized unit testing is a good thing and that making it required is really in the best interest of the 'largeish' projects we work on. I am not a developer, but I am in Quality Assurance and would like to improve the quality of the work delivered to ensure it is ready to test.
By formalized unit tests, I'm simply talking about
Identifying the Unit Tests to be written
Identifying the test data (or describe it)
Writing these tests
Tracking these tests (and re-using as needed)
Making the results available
A very convincing way is to do formalized unit test yourself, regardless of what your team/company does. This might take some extra effort on your side, especially if you're not experienced with this sort of practice.
When you can then show your code is better and you are being more productive than your fellow developers, they are going to want to know why. Then feed them your favorite unit testing methods.
Once you've convinced your fellow developers, convince management together.
I use Maven with the Surefire and Cobertura plugins for all my builds. The actual test cases are created with JUnit, DbUnit and EasyMock.
Identifying Unit Tests
I try to follow Test Driven Development but to be honest I usually just do that for the handful of the test cases and then come back and create tests for the edge and exception cases later.
Identifying Test Data
DbUnit is great for loading test data for your unit tests.
Writing Test Cases
I use JUnit to create the test cases. I try to write self documenting test cases but will use Javadocs to comment something that is not obvious.
Tracking & Making The Results Available
I integrate the unit testing into my Maven build cycle using the Surefire plugin and I use the Corbertura plugin to measure the coverage achieved by those tests. I always generate and publish a web-site including the Surefire and Cobertura reports as part of my daily build so I can see what tests failed/passed.
The event which convinced me was when we managed to regress a bug three times, in three consecutive releases. Once I realised how much more productive I was as a programmer when I wasn't constantly fixing trivial mistakes after they had gone to the client, and I could have a warm fuzzy feeling that colleagues code would do what they claimed it would, I became a convert.
Back in the day I did Cobol development on Mainframes we did this religiously in the several companies I worked in and it was accepted as the way you did things because the environment enforced it. I think it was a very typical scheme for the era and maybe some of the reasons might be applicable to you:-
Like most mainframe environments we had three realms, development, Quality Assurance and Production. Programmers developed in development and unit tested there, and once they signed off and were happy the unit was migrated to the QA environment (with the test and results docs) where it was system tested by dedicated QA staff. The development to QA migration was a formal step which happened overnight. Once QA'ed the code was migrated to Production - and we had very few bugs.
The motivation to get the unit testing done and right was that if you didn't and a bug was found by QA staff it was obvious that you hadn't done the work. Consequently your reputation depended on how rigorous you were. Of course most people would end up with the occasional bug, but coders who produced solid tested code all the time soon got a star reputation and those who produced buggy code got noticed too. The push would be always to up your game, and consequently the culture produced was one that pushed towards bug free code delivered first time.
Extracting pertinent points -
Coder reputation tied up with delivery of bug free tested code
Significant overhead associated with moving unit tested code to the next level, so motivation not to repeat this and get it right first time.
System testing performed by different people to unit testing - ideally a different team.
I'm sure your environment will differ but the principals might be translatable.
Sometimes by example is the best way. I also find that reminding people that certain things just dont happen when things are under test. Next time somebody asks you to write something, do it with tests regardless. Eventually your peers will be jealous of the ease by which you can change your code and know that it still works.
As for management you need to emphasise how much time gets wasted due to the nuclear explosion that occurs when you need to make a change to codebase X that isnt under test.
Many developers dont realise just how much they refactor without ensuring they are preserving behaviour across the entire system. For me this is the biggest benefit to unit testing and TDD in my opinion.
Software requirements change
Software changes to suit the requirements
The only certainty is change. Changing code that is not under test requires the developer to be aware of every behavioural side effect possible. The reality is that the coders who think they can read into every permutation, does so by a pain staking process of trial and error until nothing breaks obviously. At this point they check in.
The pragmatic programmer recognizes that he/she is not perfect and all knowing, and that tests are like a safety net that allows them to walk the refactoring tightrope quickly and safely.
As for when to write test on greenfield code, I'd have to advocate as much as possible. Spend the time defining the behaviours that you want out of your system and write tests initially to express those higher level constructs. Unit tests can come as thoughts crystallize.
Hope this helps.
Education and/or certification.
Give your team members a formal training in the field of testing - maybe with certification exam (depending on your team members and your own attitude towards certification). You'll take testing to a higher level that way, and your team members will be more likely to take a professional attitude towards testing.
There is a big difference between convincing and requiring.
If you find a way to convince your colleagues to write them - great. However if you create some formalized rules and require them to write unit tests, they will find a way to overcome this. As a result you will get a bunch of unit tests which are worth nothing: There will be unit test for every single class available and they will test setters and getters.
Think twice before creating and enforcing rules. Developers are good at overcoming them.
First time around you just need to go ahead and write them and show people that it's worth it. I've found on three projects that it's the only way to convince people. Some people who don't code (e.g. junior project managers) won't be able to see the value until it's staring them right in the face.
On my software team, we tend to write a small business case on these issues and present them to management in order to have the time available to create and track tests. We explain that the time taken to test is well made up for when crunch time comes and everything is on the line. We also set up a Hudson build server to centralize the tracking of the unit tests. This makes it a lot easier for the developers to keep track of failing tests and to discover recurring problems.
Remind your team or the other developers that they're professionals, not amateurs. Worked for me!
Also, it's an industry standard these days. Without unit testing experience, they are less desirable and less valuable as employees to potential future employers.
As a team lead, it is my responsibility to ensure that my programmers are doing unit testing on all the modules they work on. I suppose at this point, it's not even a question of how to convince them, it's required. Not sometimes, not on largish projects, all the time. Unit testing is the first line of defense against putting something in production that you will have to maintain. If something is put into production that has not been completely unit and system tested, then it will come back to bite you. I guess one of the policies we have here to support this is that if it blows in production, or causes problems, then the programmer responsible for coding and testing that module will be the one that has to take care of the problems, do the cleanup, etc. That alone is a fairly good motivator.
The other is that it is about pride. I work in a shop of about 75 coders, although that is large by some standards, it's really small enough for all of us to know one another. Its also small enough that we know what one another is working on, and when it does move to production, we are aware of any abends, failures, etc. If you are careful, do the unit and system testing, the chances of moving something to production without causing failures increases significantly. It may take a time or two of moving something to production and failing to realize it, but there are great rewards involved in not messing up. It's really nice to hear congratulations in the hallway when you move a project in and it doesn't screw up.
Write a bunch of them and demonstrate that unit testing has improved your productivity and the quality of your code. Without some kind of proof, sometimes people won't believe it's worth it.
So, two years after I asked this question, I find that one unexpected answer was that by moving to a new SDLC was what was needed. Five years ago, we established our first formal SDLC. It improved our situation, but left out some important things, such as automation. We are now in the process of establishing a new SDLC (under new managment) where one of the tenants is automation. Not just automated unit tests, but automated functional tests.
I guess the lesson is that I was thinking too small. If you are going to change how you create software, go 'whole hog' and make a drastic change rather than propose incremental change if you are not used to that.
You could take some inspiration from an initiative at Google. Their test team started putting up examples, tips and benefits inside the toilet cubicles to raise the profile of the merits of test automation.
https://testing.googleblog.com/2007/01/introducing-testing-on-toilet.html